PCI/TSM: TEE I/O infrastructure
Subsystems (10)
v2 cover PCI/TSM: TEE I/O infrastructure
Changes since v1 [1]:
- Add a netlink ABI for conveying device attestation evidence and interface reports
- Add a module autoprobe policy proposal
- Add simulated device evidence support to samples/devsec/
- Add MMIO resource evaluation from a TDISP device interface report
- Include device_cc_accepted() proposals for DMA setup
- Restore a lookup mechanism from tsm class device to all established streams
- Clarify TEE vs Confidential vs Private in drivers/base/coco.c (Greg)
- Move 'cc_accepted' to an additional bitfield next to 'dead' (Greg)
- Drop device_cc_probe() proposal (Jason)
[1]: http://lore.kernel.org/[email protected]
---
Overview
========
TEE I/O starts with the premise that devices are adversarial. That threat model needs a series of new ABIs and mechanisms. The x86 changes and the samples/devsec/ implementation in this set serve to have a consumer for all of these proposed mechanisms.
1/ Userspace needs to be able to intercept driver attach. If a relying
party does not endorse the system talking to a given device then userspace needs a control point to decline device operation. Module policy is suitable for that policy mechanism. See "device core:
Autoprobe considered harmful?"
2/ Userspace needs to be able to gather evidence that validates the
device's identity, configuration, and active mappings of MMIO and DMA. See "PCI/TSM: Add 'evidence' support"
3/ To gather and act on device evidence a device needs a "lock"
mechanism to hold a stable configuration, and an "accept" mechanism to bring the device into operation after relying party validation. See "PCI/TSM: Add Device Security (TVM Guest) LOCK operation support" and "PCI/TSM: Add Device Security (TVM Guest) ACCEPT operation support".
4/ Drivers must be unmodified (1): ioremap() requests must automatically
determine whether a resource range is mapped as encrypted or not. See "x86, ioremap, resource: Support IORES_DESC_ENCRYPTED for encrypted PCI MMIO". TODO: test unencrypted ranges in the middle of a PCI device BAR that is otherwise encrypted (MSI-X table case).
5/ Drivers must be unmodified (2): dma_alloc_coherent() and dma_map()
need to bypass swiotlb and potentially modify DMA handles when a device is accepted to DMA direct to private memory. See "x86, swiotlb: Teach swiotlb to skip 'accepted' devices" and "x86, dma: Allow accepted devices to map private memory".
Note an example SEV-TIO implementation of the lock+accept operations is out for review here [2] (based on older baseline of tsm.git#staging).
[2]: http://lore.kernel.org/[email protected]
On PCI/TSM Netlink and Rust SPDM
================================
The PCI/TSM netlink proposal is a result of the discussion from the Rust SPDM proposal [3]. That thread discussed the merits of an SPDM netlink ABI that multicasts signature events and a ".cma" keyring to authenticate PCI devices. The PCI/TSM netlink proposal diverges significantly based on the following assumptions:
1/ Device acceptance decisions are based on evidence material beyond
whether the device publishes a valid root certificate (kernel SPDM library proposal).
2/ Automatic device identity revalidation after reset is secondary to
initial device acceptance. It is follow-on work that can be achieved without a ".cma" ring. For example, cache a hash of the device certificate chain and / or measurements. Otherwise, mere identity revalidation is insufficient for PCI TDISP.
3/ Device evidence mutates based on userspace taking action on the
device state. For example, the device interface report is not available until post "lock". The result, the netlink interface must be on demand, not implicit multicast. PCI/TSM evidence conveyance is a netlink "dump" request.
The proposal for how the native kernel SPDM support would interact with the PCI/TSM implementation is via an "spdm-tsm" driver. An "spdm-tsm" driver allows for userspace policy to select between a kernel native "spdm-tsm" and "$platform-tsm" as only one TSM can have a session established at a time.
[3]: http://lore.kernel.org/[email protected]
On PCI/TSM Netlink and guest request
====================================
One of the open questions is whether pci_tsm_guest_req() should be used to convey device evidence to guests. In other words, if the core kernel understands 'struct pci_tsm_evidence' in a common way across architectures, why not implement a common transport and save pci_tsm_guest_req() for other ancillary messages that are indeed implementation specific?
This all passes a tools/testing/devsec/devsec.sh run. It wants a rebase on v7.0-rc2. It is pushed out as new tag, devsec-20260302, in the tsm.git#staging tree. The Maturity Map [4] has been updated accordingly.
Dan Williams (19):
PCI/TSM: Report active IDE streams per host bridge
device core: Fix kernel-doc warnings in base.h
device core: Introduce confidential device acceptance
modules: Document the global async_probe parameter
device core: Autoprobe considered harmful?
PCI/TSM: Add Device Security (TVM Guest) LOCK operation support PCI/TSM: Add Device Security (TVM Guest) ACCEPT operation support PCI/TSM: Add "evidence" support
PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report x86, swiotlb: Teach swiotlb to skip "accepted" devices x86, dma: Allow accepted devices to map private memory x86, ioremap, resource: Support IORES_DESC_ENCRYPTED for encrypted PCI
MMIO
samples/devsec: Introduce a PCI device-security bus + endpoint sample samples/devsec: Add sample IDE establishment samples/devsec: Add sample TSM bind and guest_request flows samples/devsec: Introduce a "Device Security TSM" sample driver tools/testing/devsec: Add a script to exercise samples/devsec/ samples/devsec: Add evidence support
tools/testing/devsec: Add basic evidence retrieval validation
drivers/base/Kconfig | 28 +
drivers/pci/Kconfig | 2 +
samples/Kconfig | 19 +
drivers/base/Makefile | 1 +
drivers/pci/Makefile | 2 +-
drivers/pci/tsm/Makefile | 9 +
samples/Makefile | 1 +
samples/devsec/Makefile | 16 +
Documentation/ABI/stable/sysfs-module | 20 +
Documentation/ABI/testing/sysfs-bus-pci | 47 +-
Documentation/ABI/testing/sysfs-class-tsm | 32 +
Documentation/ABI/testing/sysfs-faux-devsec | 15 +
Documentation/driver-api/pci/tsm.rst | 44 ++
Documentation/netlink/specs/pci-tsm.yaml | 151 ++++
drivers/base/base.h | 89 ++-
drivers/pci/tsm/netlink.h | 23 +
include/linux/device.h | 23 +
include/linux/ioport.h | 2 +
include/linux/module.h | 14 +
include/linux/pci-ide.h | 2 +
include/linux/pci-tsm.h | 121 ++-
include/linux/swiotlb.h | 15 +-
include/linux/tsm.h | 3 +
include/uapi/linux/pci-tsm-netlink.h | 101 +++
samples/devsec/devsec.h | 48 ++
arch/x86/kernel/pci-dma.c | 2 +-
arch/x86/mm/ioremap.c | 49 +-
arch/x86/mm/mem_encrypt.c | 5 +-
drivers/base/bus.c | 7 +-
drivers/base/coco.c | 58 ++
drivers/base/dd.c | 26 +-
drivers/pci/ide.c | 4 +
drivers/pci/{tsm.c => tsm/core.c} | 532 ++++++++++++-
drivers/pci/tsm/evidence.c | 274 +++++++
drivers/pci/tsm/netlink.c | 43 ++
drivers/virt/coco/tsm-core.c | 138 ++++
kernel/dma/swiotlb.c | 1 +
kernel/module/main.c | 13 +
samples/devsec/bus.c | 784 ++++++++++++++++++++
samples/devsec/common.c | 160 ++++
samples/devsec/link_tsm.c | 432 +++++++++++
samples/devsec/pci.c | 39 +
samples/devsec/tsm.c | 131 ++++
tools/testing/devsec/devsec.sh | 280 +++++++
MAINTAINERS | 6 +-
45 files changed, 3736 insertions(+), 76 deletions(-)
create mode 100644 drivers/pci/tsm/Makefile
create mode 100644 samples/devsec/Makefile
create mode 100644 Documentation/ABI/testing/sysfs-faux-devsec
create mode 100644 Documentation/netlink/specs/pci-tsm.yaml
create mode 100644 drivers/pci/tsm/netlink.h
create mode 100644 include/uapi/linux/pci-tsm-netlink.h
create mode 100644 samples/devsec/devsec.h
create mode 100644 drivers/base/coco.c
rename drivers/pci/{tsm.c => tsm/core.c} (63%)
create mode 100644 drivers/pci/tsm/evidence.c
create mode 100644 drivers/pci/tsm/netlink.c
create mode 100644 samples/devsec/bus.c
create mode 100644 samples/devsec/common.c
create mode 100644 samples/devsec/link_tsm.c
create mode 100644 samples/devsec/pci.c
create mode 100644 samples/devsec/tsm.c
create mode 100755 tools/testing/devsec/devsec.shbase-commit: c2012263047689e495e81c96d7d5b0586299578d
Signature · 1 line
2.52.0
PATCH v2 1/19 PCI/TSM: Report active IDE streams per host bridge
The first attempt at an ABI for this failed to account for naming collisions across host bridges:
Commit a4438f06b1db ("PCI/TSM: Report active IDE streams")
Revive this ABI with a per host bridge link that appears at first stream creation for a given host bridge and disappears after the last stream is removed.
For systems with many host bridge objects it allows:
ls /sys/class/tsm/tsmN/pci*/stream*
...to find all the host bridges with active streams without first iterating over all host bridges. Yilun notes that is handy to have this short cut [1] and from an administrator perspective it helps with inventory for constrained stream resources.
--- Documentation/ABI/testing/sysfs-class-tsm | 13 +++ include/linux/pci-ide.h | 2 + include/linux/tsm.h | 3 + drivers/pci/ide.c | 4 + drivers/virt/coco/tsm-core.c | 97 +++++++++++++++++++++++ 5 files changed, 119 insertions(+)
Documentation/ABI/testing/sysfs-class-tsm +13 -0
@@ -7,3 +7,16 @@ Description: signals when the PCI layer is able to support establishment of link encryption and other device-security features coordinated through a platform tsm. + +What: /sys/class/tsm/tsmN/pciDDDD:BB +Contact: [email protected] +Description: + (RO) When a PCIe host bridge has established a secure connection + via a TSM to an endpoint, this symlink appears. It facilitates a + TSM instance scoped view of PCIe Link Encryption and Secure + Session resource consumption across host bridges. The symlink + appears when a host bridge has 1 or more IDE streams established + with this TSM, and disappears when that number returns to 0. See + Documentation/ABI/testing/sysfs-devices-pci-host-bridge for the + description of the pciDDDD:BB/streamH.R.E symlink and the + pciDDDD:BB/available_secure_streams attribute.
include/linux/pci-ide.h +2 -0
@@ -82,6 +82,7 @@ struct pci_ide_regs {* @host_bridge_stream: allocated from host bridge @ide_stream_ida pool * @stream_id: unique Stream ID (within Partner Port pairing) * @name: name of the established Selective IDE Stream in sysfs +* @tsm_dev: For TSM established IDE, the TSM device context * * Negative @stream_id values indicate "uninitialized" on the * expectation that with TSM established IDE the TSM owns the stream_id
@@ -93,6 +94,7 @@ struct pci_ide {u8 host_bridge_stream;int stream_id;const char *name; +struct tsm_dev *tsm_dev; };/*
include/linux/tsm.h +3 -0
@@ -123,4 +123,7 @@ int tsm_report_unregister(const struct tsm_report_ops *ops);struct tsm_dev *tsm_register (struct device *parent,struct pci_tsm_ops *ops);void tsm_unregister (struct tsm_dev *tsm_dev);struct tsm_dev *find_tsm_dev (int id); +struct pci_ide ; +int tsm_ide_stream_register (struct pci_ide *ide); +void tsm_ide_stream_unregister (struct pci_ide *ide);#endif /* __TSM_H */
drivers/pci/ide.c +4 -0
@@ -11,6 +11,7 @@#include <linux/pci_regs.h> #include <linux/slab.h> #include <linux/sysfs.h> +#include <linux/tsm.h> #include "pci.h"
@@ -372,6 +373,9 @@ void pci_ide_stream_release(struct pci_ide *ide)if (ide->partner[PCI_IDE_EP ].enable)pci_ide_stream_disable (pdev, ide); +if (ide->tsm_dev) +tsm_ide_stream_unregister (ide); +if (ide->partner[PCI_IDE_RP ].setup)pci_ide_stream_teardown (rp, ide);
drivers/virt/coco/tsm-core.c +97 -0
119 lines not on this page · raw message
Signature · 1 line
2.52.0
Re: [PATCH v2 01/19] PCI/TSM: Report active IDE streams per host bridge
Quoted 26 lines · On Mon, 2 Mar 2026 16:01:49 -0800 Dan Williams <[email protected]>…
The first attempt at an ABI for this failed to account for naming collisions across host bridges:
Commit
a4438f06b1db("PCI/TSM: Report active IDE streams")Revive this ABI with a per host bridge link that appears at first stream creation for a given host bridge and disappears after the last stream is removed.
For systems with many host bridge objects it allows:
ls
/sys/class/tsm/tsmN/pci*/stream*...to find all the host bridges with active streams without first iterating over all host bridges. Yilun notes that is handy to have this short cut [1] and from an administrator perspective it helps with inventory for constrained stream resources.
Signed-off-by Dan Williams <[email protected]>drivers/virt/coco/tsm-core.c +0
-0
drivers/virt/coco/tsm-core.c line 12
Quoted 9 lines
drivers/virt/coco/tsm-core.c +8
-0@@ -4,10 +4,12 @@ + +static struct tsm_ide_stream *create_streams (struct tsm_dev *tsm_dev, +struct pci_host_bridge *bridge) +{ +int rc; + + struct tsm_ide_stream *streams __free(kfree) = +kzalloc (sizeof (*streams),GFP_KERNEL );
Crossed with kzalloc_obj() etc being introduced which seems appropriate here.
Quoted 20 lines
drivers/virt/coco/tsm-core.c +20
-0+if (!streams) +return NULL ; + + streams->tsm_dev = tsm_dev; + streams->bridge = bridge; +kref_init (&streams->kref); + rc =xa_insert (&tsm_ide_streams, (unsigned long )bridge, streams, +GFP_KERNEL ); +if (rc) +return NULL ; + + rc =sysfs_create_link (&tsm_dev->dev.kobj, &bridge->dev.kobj, +dev_name (&bridge->dev)); +if (rc) { +xa_erase (&tsm_ide_streams, (unsigned long )bridge); +return NULL ; + } + +return no_free_ptr (streams); +}
Re: [PATCH v2 01/19] PCI/TSM: Report active IDE streams per host bridge
In reply to Dan WilliamsQuoted 13 lines · Dan Williams
The first attempt at an ABI for this failed to account for naming collisions across host bridges:
Commit
a4438f06b1db("PCI/TSM: Report active IDE streams")Revive this ABI with a per host bridge link that appears at first stream creation for a given host bridge and disappears after the last stream is removed.
For systems with many host bridge objects it allows:
ls
/sys/class/tsm/tsmN/pci*/stream*
Quoted 7 lines
...to find all the host bridges with active streams without first iterating over all host bridges. Yilun notes that is handy to have this short cut [1] and from an administrator perspective it helps with inventory for constrained stream resources.
PATCH v2 2/19 device core: Fix kernel-doc warnings in base.h
In reply to Dan WilliamsIn preparation for adding new fields to 'struct device_private' fix up existing kernel-doc warnings in this header file of the form:
Warning: drivers/base/base.h:59 struct member 'subsys' not described in 'subsys_private'
Warning: drivers/base/base.h:59 struct member 'devices_kset' not described in 'subsys_private'
Warning: drivers/base/base.h:59 struct member 'interfaces' not described in 'subsys_private'
Warning: drivers/base/base.h:59 struct member 'mutex' not described in 'subsys_private'
...which are simple replacements of " - " with ": ".
Add new descriptions for these previously undescribed fields:
Warning: drivers/base/base.h:58 struct member 'drivers_autoprobe' not described in 'subsys_private'
Warning: drivers/base/base.h:117 struct member 'deferred_probe_reason' not described in 'device_private'
--- drivers/base/base.h | 79 +++++++++++++++++++++++---------------------- 1 file changed, 41 insertions(+), 38 deletions(-)
drivers/base/base.h +41 -38
@@ -13,27 +13,28 @@#include <linux/notifier.h> /** -* struct subsys_private - structure to hold the private to the driver core portions of the bus_type/class structure. -* -* @subsys - the struct kset that defines this subsystem -* @devices_kset - the subsystem's 'devices' directory -* @interfaces - list of subsystem interfaces associated -* @mutex - protect the devices, and interfaces lists. -* -* @drivers_kset - the list of drivers associated -* @klist_devices - the klist to iterate over the @devices_kset -* @klist_drivers - the klist to iterate over the @drivers_kset -* @bus_notifier - the bus notifier list for anything that cares about things -* on this bus. -* @bus - pointer back to the struct bus_type that this structure is associated -* with. +* struct subsys_private - structure to hold the private to the driver core +* portions of the bus_type/class structure. +* @subsys: the struct kset that defines this subsystem +* @devices_kset: the subsystem's 'devices' directory +* @interfaces: list of subsystem interfaces associated +* @mutex: protect the devices, and interfaces lists. +* @drivers_kset: the list of drivers associated +* @klist_devices: the klist to iterate over the @devices_kset +* @klist_drivers: the klist to iterate over the @drivers_kset +* @bus_notifier: the bus notifier list for anything that cares about things +* on this bus. +* @drivers_autoprobe: gate whether new devices are automatically attached to +* registered drivers, or new drivers automatically attach +* to existing devices. +* @bus: pointer back to the struct bus_type that this structure is associated +* with. * @dev_root: Default device to use as the parent. -* -* @glue_dirs - "glue" directory to put in-between the parent device to -* avoid namespace conflicts -* @class - pointer back to the struct class that this structure is associated -* with. -* @lock_key: Lock class key for use by the lock validator +* @glue_dirs: "glue" directory to put in-between the parent device to +* avoid namespace conflicts +* @class: pointer back to the struct class that this structure is associated +* with. +* @lock_key: Lock class key for use by the lock validator * * This structure is the one that is the actual kobject allowing struct * bus_type/class to be statically allocated safely. Nothing outside of the
@@ -98,24 +99,26 @@ struct driver_type {#endif /** -* struct device_private - structure to hold the private to the driver core portions of the device structure. -* -* @klist_children - klist containing all children of this device -* @knode_parent - node in sibling list -* @knode_driver - node in driver list -* @knode_bus - node in bus list -* @knode_class - node in class list -* @deferred_probe - entry in deferred_probe_list which is used to retry the -* binding of drivers which were unable to get all the resources needed by -* the device; typically because it depends on another driver getting -* probed first. -* @async_driver - pointer to device driver awaiting probe via async_probe -* @device - pointer back to the struct device that this structure is -* associated with. -* @driver_type - The type of the bound Rust driver. -* @dead - This device is currently either in the process of or has been -* removed from the system. Any asynchronous events scheduled for this -* device should exit without taking any action. +* struct device_private - structure to hold the private to the driver core +* portions of the device structure. +* @klist_children: klist containing all children of this device +* @knode_parent: node in sibling list +* @knode_driver: node in driver list +* @knode_bus: node in bus list +* @knode_class: node in class list +* @deferred_probe: entry in deferred_probe_list which is used to retry the +* binding of drivers which were unable to get all the +* resources needed by the device; typically because it depends +* on another driver getting probed first. +* @async_driver: pointer to device driver awaiting probe via async_probe +* @deferred_probe_reason: capture the -EPROBE_DEFER message emitted with +* dev_err_probe() for later retrieval via debugfs +* @device: pointer back to the struct device that this structure is +* associated with. +* @driver_type: The type of the bound Rust driver. +* @dead: This device is currently either in the process of or has been +* removed from the system. Any asynchronous events scheduled for this +* device should exit without taking any action. * * Nothing outside of the driver core should ever touch these fields. */
Signature · 1 line
2.52.0
Re: [PATCH v2 02/19] device core: Fix kernel-doc warnings in base.h
Quoted 14 lines · On Mon, 2 Mar 2026 16:01:50 -0800 Dan Williams <[email protected]>…
In preparation for adding new fields to 'struct
device_private' fix up existing kernel-doc warnings in this header file of the form:Warning:
drivers/base/base.h:59struct member 'subsys' not described in 'subsys_private'
Warning:drivers/base/base.h:59struct member 'devices_kset' not described in 'subsys_private'
Warning:drivers/base/base.h:59struct member 'interfaces' not described in 'subsys_private'
Warning:drivers/base/base.h:59struct member 'mutex' not described in 'subsys_private'...which are simple replacements of " - " with ": ".
Quoted 8 lines
Add new descriptions for these previously undescribed fields:
Warning:
drivers/base/base.h:58struct member 'drivers_autoprobe' not described in 'subsys_private'
Warning:drivers/base/base.h:117struct member 'deferred_probe_reason' not described in 'device_private'Signed-off-by Dan Williams <[email protected]>
Maybe if the rest looks 'slow' can send this one ahead?
Jonathan
Greg KH
maintainerRe: [PATCH v2 02/19] device core: Fix kernel-doc warnings in base.h
Quoted 24 lines · Jonathan Cameron
Quoted 21 lines · On Mon, 2 Mar 2026 16:01:50 -0800 Dan Williams <[email protected]>…
In preparation for adding new fields to 'struct
device_private' fix up existing kernel-doc warnings in this header file of the form:Warning:
drivers/base/base.h:59struct member 'subsys' not described in 'subsys_private'
Warning:drivers/base/base.h:59struct member 'devices_kset' not described in 'subsys_private'
Warning:drivers/base/base.h:59struct member 'interfaces' not described in 'subsys_private'
Warning:drivers/base/base.h:59struct member 'mutex' not described in 'subsys_private'...which are simple replacements of " - " with ": ".
Add new descriptions for these previously undescribed fields:
Warning:
drivers/base/base.h:58struct member 'drivers_autoprobe' not described in 'subsys_private'
Warning:drivers/base/base.h:117struct member 'deferred_probe_reason' not described in 'device_private'
Quoted 5 lines
Quoted 1 line
Signed-off-by Dan Williams <[email protected]>Maybe if the rest looks 'slow' can send this one ahead?
Reviewed-by Jonathan Cameron <[email protected]>
I can takt this now, thanks.
greg k-h
PATCH v2 3/19 device core: Introduce confidential device acceptance
In reply to Dan WilliamsAn "accepted" device is one that is allowed to access private memory within a Trusted Computing Boundary (TCB). The concept of "acceptance" is distinct from other device properties like usb_device::authorized, or tb_switch::authorized. The entry to the accepted state is a violent operation in which the device will reject MMIO requests that are not encrypted, and the device enters a new IOMMU protection domain to allow it to access addresses that were previously off-limits.
Subsystems like the DMA mapping layer, that need to modify their behavior based on the accept state, may only have access to the base 'struct device'. It is also likely that the concept of TCB acceptance grows beyond PCI devices over time. For these reasons, introduce the concept of acceptance in 'struct device_private' which is device common, but only suitable for buses and built-in infrastructure to consume.
--- drivers/base/Kconfig | 4 +++ drivers/base/Makefile | 1 + drivers/base/base.h | 9 +++++++ include/linux/device.h | 22 ++++++++++++++++ drivers/base/coco.c | 58 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 94 insertions(+) create mode 100644 drivers/base/coco.c
drivers/base/Kconfig +4 -0
@@ -249,4 +249,8 @@ config FW_DEVLINK_SYNC_STATE_TIMEOUT command line option on every system/board your kernel is expected to work on. +config CONFIDENTIAL_DEVICES + depends on ARCH_HAS_CC_PLATFORM + bool + endmenu
drivers/base/Makefile +1 -0
@@ -27,6 +27,7 @@ obj-$(CONFIG_GENERIC_MSI_IRQ) += platform-msi.o obj-$(CONFIG_GENERIC_ARCH_TOPOLOGY) += arch_topology.o obj-$(CONFIG_GENERIC_ARCH_NUMA) += arch_numa.o obj-$(CONFIG_ACPI) += physical_location.o +obj-$(CONFIG_CONFIDENTIAL_DEVICES) += coco.o obj-y += test/
drivers/base/base.h +9 -0
@@ -119,8 +119,13 @@ struct driver_type {* @dead: This device is currently either in the process of or has been * removed from the system. Any asynchronous events scheduled for this * device should exit without taking any action. +* @cc_accepted: track the TEE acceptance state of the device for deferred +* probing, MMIO mapping type, and SWIOTLB bypass for private memory DMA. * * Nothing outside of the driver core should ever touch these fields. +* +* All bitfield flags are manipulated under device_lock() to avoid +* read-modify-write collisions. */ struct device_private {struct klist klist_children;
@@ -136,6 +141,10 @@ struct device_private {struct driver_type driver_type;#endif u8 dead:1; +#ifdef CONFIG_CONFIDENTIAL_DEVICES + u8 cc_accepted:1; +#endif + };#define to_device_private_parent (obj) \ container_of(obj, struct device_private, knode_parent)
include/linux/device.h +22 -0
@@ -1191,6 +1191,28 @@ static inline bool device_link_test(const struct device_link *link, u32 flags)return !!(link->flags & flags); } +/* Confidential Device state helpers */ +#ifdef CONFIG_CONFIDENTIAL_DEVICES +int device_cc_accept (struct device *dev); +int device_cc_reject (struct device *dev); +bool device_cc_accepted (struct device *dev); +#else +static inline int device_cc_accept (struct device *dev) +{ +lockdep_assert_held (&dev->mutex); +return 0 ; +} +static inline int device_cc_reject (struct device *dev) +{ +lockdep_assert_held (&dev->mutex); +return 0 ; +} +static inline bool device_cc_accepted (struct device *dev) +{ +return false; +} +#endif /* CONFIG_CONFIDENTIAL_DEVICES */ +/* Create alias, so I can be autoloaded. */ #define MODULE_ALIAS_CHARDEV (major,minor) \ MODULE_ALIAS("char-major-" __stringify(major) "-" __stringify(minor))
drivers/base/coco.c new file +58 -0
58 lines not on this page · raw message
Signature · 1 line
2.52.0
Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 15 lines · On Mon, 2 Mar 2026 16:01:51 -0800 Dan Williams <[email protected]>…
An "accepted" device is one that is allowed to access private memory within a Trusted Computing Boundary (TCB). The concept of "acceptance" is distinct from other device properties like
usb_device::authorized, ortb_switch::authorized. The entry to the accepted state is a violent operation in which the device will reject MMIO requests that are not encrypted, and the device enters a new IOMMU protection domain to allow it to access addresses that were previously off-limits.Subsystems like the DMA mapping layer, that need to modify their behavior based on the accept state, may only have access to the base 'struct device'. It is also likely that the concept of TCB acceptance grows beyond PCI devices over time. For these reasons, introduce the concept of acceptance in 'struct
device_private' which is device common, but only suitable for buses and built-in infrastructure to consume.
Quoted 14 lines
Cc Christoph Hellwig <[email protected]>Cc Jason Gunthorpe <[email protected]>Cc Marek Szyprowski <[email protected]>Cc Robin Murphy <[email protected]>Cc Roman Kisel <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Samuel Ortiz <[email protected]>Cc Alexey Kardashevskiy <[email protected]>Cc Xu Yilun <[email protected]>Cc "Aneesh Kumar K.V" <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc "Rafael J. Wysocki" <[email protected]>Cc Danilo Krummrich <[email protected]>Signed-off-by Dan Williams <[email protected]>
Seems reasonable to me.
Greg KH
maintainerRe: [PATCH v2 03/19] device core: Introduce confidential device acceptance
In reply to Dan WilliamsQuoted 7 lines · Dan Williams
An "accepted" device is one that is allowed to access private memory within a Trusted Computing Boundary (TCB). The concept of "acceptance" is distinct from other device properties like
usb_device::authorized, ortb_switch::authorized. The entry to the accepted state is a violent operation in which the device will reject MMIO requests that are not encrypted, and the device enters a new IOMMU protection domain to allow it to access addresses that were previously off-limits.
Trying to mix/match "acceptance" with "authorized" is going to be a nightmare, what's the combination that can happen here over time?
We need to either "trust" or "not trust" the device, and the bus can decide what to do with that value (if anything). The DMA layer can then use that value to do:
Quoted 3 lines
Subsystems like the DMA mapping layer, that need to modify their behavior based on the accept state, may only have access to the base 'struct device'.
^this.
Quoted 4 lines
It is also likely that the concept of TCB acceptance grows beyond PCI devices over time. For these reasons, introduce the concept of acceptance in 'struct
device_private' which is device common, but only suitable for buses and built-in infrastructure to consume.
Busses are what can control this, but please, let's not make this a cc-only type thing. We have the idea of trust starting to propagate through a number of different busses, let's get it right here, so we don't have to have all of these different bus-specific hacks like we do today.
Quoted 67 lines
Cc Christoph Hellwig <[email protected]>Cc Jason Gunthorpe <[email protected]>Cc Marek Szyprowski <[email protected]>Cc Robin Murphy <[email protected]>Cc Roman Kisel <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Samuel Ortiz <[email protected]>Cc Alexey Kardashevskiy <[email protected]>Cc Xu Yilun <[email protected]>Cc "Aneesh Kumar K.V" <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc "Rafael J. Wysocki" <[email protected]>Cc Danilo Krummrich <[email protected]>Signed-off-by Dan Williams <[email protected]>--- drivers/base/Kconfig | 4 +++ drivers/base/Makefile | 1 + drivers/base/base.h | 9 +++++++ include/linux/device.h | 22 ++++++++++++++++ drivers/base/coco.c | 58 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 94 insertions(+) create mode 100644 drivers/base/coco.cdrivers/base/Kconfig +4
-0@@ -249,4 +249,8 @@ config FW_DEVLINK_SYNC_STATE_TIMEOUT command line option on every system/board your kernel is expected to work on. +config CONFIDENTIAL_DEVICES + depends on ARCH_HAS_CC_PLATFORM + bool + endmenudrivers/base/Makefile +1
-0@@ -27,6 +27,7 @@ obj-$(CONFIG_GENERIC_MSI_IRQ) += platform-msi.o obj-$(CONFIG_GENERIC_ARCH_TOPOLOGY) += arch_topology.o obj-$(CONFIG_GENERIC_ARCH_NUMA) += arch_numa.o obj-$(CONFIG_ACPI) += physical_location.o +obj-$(CONFIG_CONFIDENTIAL_DEVICES) += coco.o obj-y += test/drivers/base/base.h +5
-0@@ -119,8 +119,13 @@ struct driver_type {* @dead: This device is currently either in the process of or has been * removed from the system. Any asynchronous events scheduled for this * device should exit without taking any action. +* @cc_accepted: track the TEE acceptance state of the device for deferred +* probing, MMIO mapping type, and SWIOTLB bypass for private memory DMA. * * Nothing outside of the driver core should ever touch these fields. +* +* All bitfield flags are manipulated under device_lock() to avoid +* read-modify-write collisions. */ struct device_private {struct klist klist_children;
drivers/base/base.h struct device_private { · line 146
Quoted 6 lines
drivers/base/base.h +3
-0@@ -136,6 +141,10 @@ struct device_private {struct driver_type driver_type;#endif u8 dead:1; +#ifdef CONFIG_CONFIDENTIAL_DEVICES + u8 cc_accepted:1; +#endif
Just make this:
u8 trusted:1;
no need for an #ifdef.
Quoted 9 lines
drivers/base/base.h +1
-0+ };#define to_device_private_parent (obj) \ container_of(obj, struct device_private, knode_parent)include/linux/device.h +0
-0
include/linux/device.h static inline bool device_link_test(const struct device_link *link, u32 flags) · line 1200
Quoted 10 lines
include/linux/device.h +7
-0@@ -1191,6 +1191,28 @@ static inline bool device_link_test(const struct device_link *link, u32 flags)return !!(link->flags & flags); } +/* Confidential Device state helpers */ +#ifdef CONFIG_CONFIDENTIAL_DEVICES +int device_cc_accept (struct device *dev); +int device_cc_reject (struct device *dev); +bool device_cc_accepted (struct device *dev); +#else +static inline int device_cc_accept (struct device *dev)
No __must_hold() usage? That's best to check this at build time, not just relying on:
include/linux/device.h static inline bool device_link_test(const struct device_link *link, u32 flags) · line 1202
Quoted 2 lines
include/linux/device.h +2
-0+{ +lockdep_assert_held (&dev->mutex);
runtime checks.
Same for all the calls here.
Quoted 7 lines
include/linux/device.h +7
-0+/** +* device_cc_accept(): Mark a device as able to access private memory +* @dev: device to accept +* +* Confidential bus drivers use this helper to accept devices. For example, PCI +* has a sysfs ABI to accept devices after relying party attestation. +*
include/linux/device.h static inline bool device_link_test(const struct device_link *link, u32 flags)
Quoted 10 lines
include/linux/device.h +10
-0+* Given that moving a device into confidential / private operation implicates +* changes to MMIO mapping attributes and DMA mappings, the transition must be +* done while the device is idle (driver detached). +*/ +int device_cc_accept (struct device *dev) +{ +lockdep_assert_held (&dev->mutex); + +if (dev->driver) +return -EBUSY ;
So you are saying that once a driver is bound, it is "trusted"? That's fine, but maybe you don't want to do that in the core, shouldn't that be a bus-specific thing?
this could then be:
int device_trust(struct device *dev);
int device_untrust(struct device *dev); /* ugh, bad name, pick something else? */ bool device_trusted(struct device *dev);
but note, do you ever want to move a device from trusted to untrusted? What would cause that?
thanks,
greg k-h
Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 11 lines · Greg KH
Quoted 7 lines · Dan Williams
An "accepted" device is one that is allowed to access private memory within a Trusted Computing Boundary (TCB). The concept of "acceptance" is distinct from other device properties like
usb_device::authorized, ortb_switch::authorized. The entry to the accepted state is a violent operation in which the device will reject MMIO requests that are not encrypted, and the device enters a new IOMMU protection domain to allow it to access addresses that were previously off-limits.Trying to mix/match "acceptance" with "authorized" is going to be a nightmare, what's the combination that can happen here over time?
I do think Linux needs to mix/match these concepts. "Authorization" is a kernel policy to operate a device at all. "Acceptance" is a mechanism to operate a device within a hardware TCB boundary.
So, the truth table of combinations would be:
accepted authorized result
0 0 logically and physically disconnected device
0 1 connected device, DMA is bounce buffered
and loses confidentiality, integrity, and performance
1 0 logically disconnected device, but a
relying party trusts that the device is not spoofing authorization.
1 1 connected device, DMA is direct and gains
confidentiality, integrity and performance
To say it another way, when the above distinguishes "logically" vs "physically" disconnected it is whether the device interface can be verified to not be under adversarial control. An unaccepted device can do limited damage, but still can bounce buffer secrets out of the TCB if so directed.
Quoted 3 lines
We need to either "trust" or "not trust" the device, and the bus can decide what to do with that value (if anything). The DMA layer can then use that value to do:
Trust is separate. For example, there are deployed use cases today where the device is trusted, but unaccepted. Acceptance support for those cases is mostly a performance optimization to be able to stop performing software encryption on top of DMA bounce buffering.
Quoted 5 lines
Quoted 3 lines
Subsystems like the DMA mapping layer, that need to modify their behavior based on the accept state, may only have access to the base 'struct device'.
^this.
The DMA layer is not operating on a trust concept it is effectively being told to select an IOMMU.
Quoted 10 lines
Quoted 4 lines
It is also likely that the concept of TCB acceptance grows beyond PCI devices over time. For these reasons, introduce the concept of acceptance in 'struct
device_private' which is device common, but only suitable for buses and built-in infrastructure to consume.Busses are what can control this, but please, let's not make this a cc-only type thing. We have the idea of trust starting to propagate through a number of different busses, let's get it right here, so we don't have to have all of these different bus-specific hacks like we do today.
The conflation of "trust" and "acceptance" has been the main stumbling block of past proposals. As you have said before "kernel drivers trust their devices". That precedent is not being touched in this proposal. Instead, give userspace all the tools it needs to deploy policy about when to operate a device. When it does decide to operate the device give it the mechanism to add confidentiality, integrity and performance to that operation.
This is a "CC-only type thing" because only CC partitions the system into two device domains. One where "trusted unaccepted" devices can operate without CC protections and "trusted accepted" devices can operate with CC protections and direct DMA.
Quoted 76 lines
Quoted 68 lines
Cc Christoph Hellwig <[email protected]>Cc Jason Gunthorpe <[email protected]>Cc Marek Szyprowski <[email protected]>Cc Robin Murphy <[email protected]>Cc Roman Kisel <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Samuel Ortiz <[email protected]>Cc Alexey Kardashevskiy <[email protected]>Cc Xu Yilun <[email protected]>Cc "Aneesh Kumar K.V" <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc "Rafael J. Wysocki" <[email protected]>Cc Danilo Krummrich <[email protected]>Signed-off-by Dan Williams <[email protected]>--- drivers/base/Kconfig | 4 +++ drivers/base/Makefile | 1 + drivers/base/base.h | 9 +++++++ include/linux/device.h | 22 ++++++++++++++++ drivers/base/coco.c | 58 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 94 insertions(+) create mode 100644 drivers/base/coco.cdrivers/base/Kconfig +4
-0@@ -249,4 +249,8 @@ config FW_DEVLINK_SYNC_STATE_TIMEOUT command line option on every system/board your kernel is expected to work on. +config CONFIDENTIAL_DEVICES + depends on ARCH_HAS_CC_PLATFORM + bool + endmenudrivers/base/Makefile +1
-0@@ -27,6 +27,7 @@ obj-$(CONFIG_GENERIC_MSI_IRQ) += platform-msi.o obj-$(CONFIG_GENERIC_ARCH_TOPOLOGY) += arch_topology.o obj-$(CONFIG_GENERIC_ARCH_NUMA) += arch_numa.o obj-$(CONFIG_ACPI) += physical_location.o +obj-$(CONFIG_CONFIDENTIAL_DEVICES) += coco.o obj-y += test/drivers/base/base.h +5
-0@@ -119,8 +119,13 @@ struct driver_type {* @dead: This device is currently either in the process of or has been * removed from the system. Any asynchronous events scheduled for this * device should exit without taking any action. +* @cc_accepted: track the TEE acceptance state of the device for deferred +* probing, MMIO mapping type, and SWIOTLB bypass for private memory DMA. * * Nothing outside of the driver core should ever touch these fields. +* +* All bitfield flags are manipulated under device_lock() to avoid +* read-modify-write collisions. */ struct device_private {struct klist klist_children;drivers/base/base.h struct device_private { · line 146
Quoted 6 lines
drivers/base/base.h +3
-0@@ -136,6 +141,10 @@ struct device_private {struct driver_type driver_type;#endif u8 dead:1; +#ifdef CONFIG_CONFIDENTIAL_DEVICES + u8 cc_accepted:1; +#endif Just make this:
Quoted 3 lines
u8 trusted:1;no need for an #ifdef.
Ok.
Quoted 25 lines
Quoted 11 lines
drivers/base/base.h +1
-0+ };#define to_device_private_parent (obj) \ container_of(obj, struct device_private, knode_parent)include/linux/device.h +0
-0include/linux/device.h static inline bool device_link_test(const struct device_link *link, u32 flags) · line 1200
Quoted 10 lines
include/linux/device.h +7
-0@@ -1191,6 +1191,28 @@ static inline bool device_link_test(const struct device_link *link, u32 flags)return !!(link->flags & flags); } +/* Confidential Device state helpers */ +#ifdef CONFIG_CONFIDENTIAL_DEVICES +int device_cc_accept (struct device *dev); +int device_cc_reject (struct device *dev); +bool device_cc_accepted (struct device *dev); +#else +static inline int device_cc_accept (struct device *dev)No
__must_hold()usage? That's best to check this at build time, not just relying on:
Quoted 6 lines
include/linux/device.h static inline bool device_link_test(const struct device_link *link, u32 flags) · line 1202
Quoted 2 lines
include/linux/device.h +2
-0+{ +lockdep_assert_held (&dev->mutex);runtime checks.
Same for all the calls here.
Ok, will take a look and convert.
Quoted 13 lines
Quoted 13 lines
include/linux/device.h +13
-0+/** +* device_cc_accept(): Mark a device as able to access private memory +* @dev: device to accept +* +* Confidential bus drivers use this helper to accept devices. For example, PCI +* has a sysfs ABI to accept devices after relying party attestation. +* +* Given that moving a device into confidential / private operation implicates +* changes to MMIO mapping attributes and DMA mappings, the transition must be +* done while the device is idle (driver detached). +*/ +int device_cc_accept (struct device *dev) +{
Quoted 8 lines
include/linux/device.h static inline bool device_link_test(const struct device_link *link, u32 flags)
Quoted 4 lines
include/linux/device.h +4
-0+lockdep_assert_held (&dev->mutex); + +if (dev->driver) +return -EBUSY ;So you are saying that once a driver is bound, it is "trusted"? That's fine, but maybe you don't want to do that in the core, shouldn't that be a bus-specific thing?
No, per above this not about trust. This is about the fact that the device can not switch between DMA/IOMMU and encrypted MMIO operational models while it may have active DMA or MMIO mappings. So the check here is to observe that the mechanism of toggling inside / outside TCB operation is so violent as to not be something that can change while the device is being operated by a driver.
Quoted 8 lines
this could then be:
intdevice_trust(struct device *dev);
intdevice_untrust(struct device *dev); /* ugh, bad name, pick something else? */ booldevice_trusted(struct device *dev);but note, do you ever want to move a device from trusted to untrusted? What would cause that?
Moving to unaccepted operation is when any detail that the relying party used to make "accept" decision changed. For example, the device moves to the ERROR security state from severe events like PCIe link encryption loss, to modest events like the host VMM toggled the bus-master-enable bit in the physical function's command register.
So error recovery takes the device from RUN to ERROR before it can be transitioned back to LOCK then RUN (accepted). But also a coordinated firmware update would need to take the device from RUN to UNLOCKED to LOCKED (revalidate device evidence) back to RUN.
Greg KH
maintainerRe: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 30 lines · Dan Williams
Quoted 11 lines · Greg KH
Quoted 7 lines · Dan Williams
An "accepted" device is one that is allowed to access private memory within a Trusted Computing Boundary (TCB). The concept of "acceptance" is distinct from other device properties like
usb_device::authorized, ortb_switch::authorized. The entry to the accepted state is a violent operation in which the device will reject MMIO requests that are not encrypted, and the device enters a new IOMMU protection domain to allow it to access addresses that were previously off-limits.Trying to mix/match "acceptance" with "authorized" is going to be a nightmare, what's the combination that can happen here over time?
I do think Linux needs to mix/match these concepts. "Authorization" is a kernel policy to operate a device at all. "Acceptance" is a mechanism to operate a device within a hardware TCB boundary.
So, the truth table of combinations would be:
accepted authorized result
0 0 logically and physically disconnected device0 1 connected device, DMA is bounce buffered
and loses confidentiality, integrity, and performance1 0 logically disconnected device, but a
relying party trusts that the device is not spoofing authorization.
Quoted 8 lines
1 1 connected device, DMA is direct and gains
confidentiality, integrity and performanceTo say it another way, when the above distinguishes "logically" vs "physically" disconnected it is whether the device interface can be verified to not be under adversarial control. An unaccepted device can do limited damage, but still can bounce buffer secrets out of the TCB if so directed.
I really don't agree with this, but I can't think of why at the moment. I feel like you are looking at this purely in the TCB point of view, while I don't feel that is something that should be considered "special" at all here. Linux has, for the most part, always trusted the hardware, and now you are wanting to not trust the hardware for some things and parts of the kernel. Which is great, it's something that I have wanted to change for a very long time now, but let's do it right if at all possible.
Give me a few days to come up with a better reply, let me think about this some more...
Quoted 8 lines
Quoted 3 lines
We need to either "trust" or "not trust" the device, and the bus can decide what to do with that value (if anything). The DMA layer can then use that value to do:
Trust is separate. For example, there are deployed use cases today where the device is trusted, but unaccepted. Acceptance support for those cases is mostly a performance optimization to be able to stop performing software encryption on top of DMA bounce buffering.
If "acceptance" is just a performance issue, I think you all need to go back to the marketing people as that's probably not what they intended to have happen here. For some reason I thought they were selling this as "security", not "speed" :)
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Subsystems like the DMA mapping layer, that need to modify their behavior based on the accept state, may only have access to the base 'struct device'.
^this.
The DMA layer is not operating on a trust concept it is effectively being told to select an IOMMU.
Ok, then that's independent of "acceptance", that is "use this IOMMU vs. that one" type of thing which is just a "basic configuration for speed" type of thing as you mention above :)
Let's not confuse that with anything else like "acceptance" please.
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
It is also likely that the concept of TCB acceptance grows beyond PCI devices over time. For these reasons, introduce the concept of acceptance in 'struct
device_private' which is device common, but only suitable for buses and built-in infrastructure to consume.Busses are what can control this, but please, let's not make this a cc-only type thing. We have the idea of trust starting to propagate through a number of different busses, let's get it right here, so we don't have to have all of these different bus-specific hacks like we do today.
Quoted 3 lines
The conflation of "trust" and "acceptance" has been the main stumbling block of past proposals. As you have said before "kernel drivers trust their devices". That precedent is not being touched in this proposal.
Ah, but I WANT to touch that. Let's FINALLY solve that! Or at the very least, provide the infrastructure in the driver core to allow busses that want to do that, to be able to do so.
Quoted 4 lines
Instead, give userspace all the tools it needs to deploy policy about when to operate a device. When it does decide to operate the device give it the mechanism to add confidentiality, integrity and performance to that operation.
Yes, this is a policy decision, and if you are only saying this is about "which IOMMU should we select", then that's a dma layer configuration option. Let's not call that "acceptance" please.
Quoted 4 lines
This is a "CC-only type thing" because only CC partitions the system into two device domains. One where "trusted unaccepted" devices can operate without CC protections and "trusted accepted" devices can operate with CC protections and direct DMA.
In other words, it's an IOMMU switch, so why not use the switch infrastructure? </me runs away...>
anyway, let me think about this some more...
thanks,
greg k-h
Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 40 lines · Greg KH
Quoted 30 lines · Dan Williams
Quoted 11 lines · Greg KH
Quoted 7 lines · Dan Williams
An "accepted" device is one that is allowed to access private memory within a Trusted Computing Boundary (TCB). The concept of "acceptance" is distinct from other device properties like
usb_device::authorized, ortb_switch::authorized. The entry to the accepted state is a violent operation in which the device will reject MMIO requests that are not encrypted, and the device enters a new IOMMU protection domain to allow it to access addresses that were previously off-limits.Trying to mix/match "acceptance" with "authorized" is going to be a nightmare, what's the combination that can happen here over time?
I do think Linux needs to mix/match these concepts. "Authorization" is a kernel policy to operate a device at all. "Acceptance" is a mechanism to operate a device within a hardware TCB boundary.
So, the truth table of combinations would be:
accepted authorized result
0 0 logically and physically disconnected device0 1 connected device, DMA is bounce buffered
and loses confidentiality, integrity, and performance1 0 logically disconnected device, but a
relying party trusts that the device is not spoofing authorization.Quoted 8 lines
1 1 connected device, DMA is direct and gains
confidentiality, integrity and performanceTo say it another way, when the above distinguishes "logically" vs "physically" disconnected it is whether the device interface can be verified to not be under adversarial control. An unaccepted device can do limited damage, but still can bounce buffer secrets out of the TCB if so directed.
Quoted 5 lines
I really don't agree with this, but I can't think of why at the moment. I feel like you are looking at this purely in the TCB point of view, while I don't feel that is something that should be considered "special" at all here. Linux has, for the most part, always trusted the hardware, and now you are wanting to not trust the hardware.
I think framing "trust" as an enum rather than a boolean better addresses this problem.
Quoted 6 lines
for some things and parts of the kernel. Which is great, it's something that I have wanted to change for a very long time now, but let's do it right if at all possible.
Give me a few days to come up with a better reply, let me think about this some more...
Sure, but I also think we might already be converging, more below...
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
We need to either "trust" or "not trust" the device, and the bus can decide what to do with that value (if anything). The DMA layer can then use that value to do:
Trust is separate. For example, there are deployed use cases today where the device is trusted, but unaccepted. Acceptance support for those cases is mostly a performance optimization to be able to stop performing software encryption on top of DMA bounce buffering.
Quoted 4 lines
If "acceptance" is just a performance issue, I think you all need to go back to the marketing people as that's probably not what they intended to have happen here. For some reason I thought they were selling this as "security", not "speed" :)
Do not get me wrong there are several threat models mitigated by having hardware assurance that all communications with the device are confidentiality and integrity protected. Hardware assurances that attempts to subvert those protections result in hardware error states is part of the value. However, you can approximate a subset of those protections with high overhead workarounds.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Subsystems like the DMA mapping layer, that need to modify their behavior based on the accept state, may only have access to the base 'struct device'.
^this.
Quoted 8 lines
Quoted 2 lines
The DMA layer is not operating on a trust concept it is effectively being told to select an IOMMU.
Ok, then that's independent of "acceptance", that is "use this IOMMU vs. that one" type of thing which is just a "basic configuration for speed" type of thing as you mention above :)
Let's not confuse that with anything else like "acceptance" please.
That is fine, and I think you and Jason may be hitting on the same concern.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
It is also likely that the concept of TCB acceptance grows beyond PCI devices over time. For these reasons, introduce the concept of acceptance in 'struct
device_private' which is device common, but only suitable for buses and built-in infrastructure to consume.Busses are what can control this, but please, let's not make this a cc-only type thing. We have the idea of trust starting to propagate through a number of different busses, let's get it right here, so we don't have to have all of these different bus-specific hacks like we do today.
Quoted 7 lines
Quoted 3 lines
The conflation of "trust" and "acceptance" has been the main stumbling block of past proposals. As you have said before "kernel drivers trust their devices". That precedent is not being touched in this proposal.
Ah, but I WANT to touch that. Let's FINALLY solve that! Or at the very least, provide the infrastructure in the driver core to allow busses that want to do that, to be able to do so.
Jason's framing of an enum rather than a boolean for "trust" seems workable to me and melds "authorization" and "CC acceptance" into one concept.
Quoted 8 lines
Quoted 4 lines
Instead, give userspace all the tools it needs to deploy policy about when to operate a device. When it does decide to operate the device give it the mechanism to add confidentiality, integrity and performance to that operation.
Yes, this is a policy decision, and if you are only saying this is about "which IOMMU should we select", then that's a dma layer configuration option. Let's not call that "acceptance" please.
Done.
...and there is precedent for a "trust" enum, lockdown levels. In that case as well there are a menu of priveleges that can be incrementally enabled by a policy.
Quoted 9 lines
Quoted 4 lines
This is a "CC-only type thing" because only CC partitions the system into two device domains. One where "trusted unaccepted" devices can operate without CC protections and "trusted accepted" devices can operate with CC protections and direct DMA.
In other words, it's an IOMMU switch, so why not use the switch infrastructure? </me runs away...>
anyway, let me think about this some more...
Will do, however in the meantime I am going to speculate that this "trust as enum" idea is workable and start drafting some patches towards that.
Jason Gunthorpe
maintainerRe: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 3 lines · Dan Williams
Jason's framing of an enum rather than a boolean for "trust" seems workable to me and melds "authorization" and "CC acceptance" into one concept.
I think you can also fold the auto-probe into this as well.
The kernel would have some default policy for what enum value to set upon discovery and instead of 'disable auto probe' you'd arrange to set trust level 0 which would block driver binding and probing inherently.
Policy in userspace then has to increase the trust level which could trigger an auto-bind.
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines
Instead, give userspace all the tools it needs to deploy policy about when to operate a device. When it does decide to operate the device give it the mechanism to add confidentiality, integrity and performance to that operation.
Yes, this is a policy decision, and if you are only saying this is about "which IOMMU should we select", then that's a dma layer configuration option. Let's not call that "acceptance" please.
Done.
AFAICT "which IOMMU should we select" should entirely be driven by the TDISP state being in RUN
Jason
Jason Gunthorpe
maintainerRe: [PATCH v2 03/19] device core: Introduce confidential device acceptance
In reply to Dan WilliamsQuoted 10 lines · Dan Williams
Quoted 9 lines · Greg KH
Quoted 7 lines · Dan Williams
An "accepted" device is one that is allowed to access private memory within a Trusted Computing Boundary (TCB). The concept of "acceptance" is distinct from other device properties like
usb_device::authorized, ortb_switch::authorized. The entry to the accepted state is a violent operation in which the device will reject MMIO requests that are not encrypted, and the device enters a new IOMMU protection domain to allow it to access addresses that were previously off-limits.
Quoted 6 lines
Quoted 2 lines
Trying to mix/match "acceptance" with "authorized" is going to be a nightmare, what's the combination that can happen here over time?
I do think Linux needs to mix/match these concepts. "Authorization" is a kernel policy to operate a device at all. "Acceptance" is a mechanism to operate a device within a hardware TCB boundary.
I'm not sure about these words either, I would revise your table to be more OS centric, the device can be in one of four security levels:
0 Blocked and disabled
The device cannot attack the system, enforced by the OS not loading a driver or mapping the MMIO and IOMMU fully blocking everything from it.
1 In use, attacks from a hostile device are possible
A driver can operate the device and is expected to defend against attacks from the device itself. The IOMMU restricts the device to only access driver approved data (no ATS, DMA strict only, CC shared only, interrupt remapping security, bounce partial DMA mappings, etc)
2 In use, no attacks from the device
The device does what the driver says and is not hostile. The driver does not have to defend itself, the IOMMU can run in faster & lower security modes (ATS on, DMA-FQ, Identity, still CC shared only)
* Basically our default security level today
3 In use, no attacks, and access to CC private memory
Like #2 and now the IOMMU allows access to CC private memory too.
[*] I'm inclunding all attacks with "hostile device", including MIM on the PCIe link, compromised/fake device, attacks from a VMM through a virtual device, etc.
From a CC VM perspective 0 is at boot, 1 is an out of TCB device, 2 doesn't exist (without TDISP there is no way to keep the hypervisor from attacking?), and 3 is a full accepted TDSIP device.
#2 can happen in bare metal where a OS may activate link encryption and attest the device, but doesn't have CC private/shared memory.
From a uAPI perspective I'm not sold on having two bools, I think a level string would be more flexible. TSM and CC properties are orthogonal, except you can't select #3 without the TSM saying it is in RUN.
Internally we'd probably turn that dev->trusted thing into an enum and teach the iommu layer to treat it more dynamically.
Jason
Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 14 lines · Jason Gunthorpe
Quoted 13 lines · Dan Williams
Quoted 11 lines · Greg KH
Quoted 7 lines · Dan Williams
An "accepted" device is one that is allowed to access private memory within a Trusted Computing Boundary (TCB). The concept of "acceptance" is distinct from other device properties like
usb_device::authorized, ortb_switch::authorized. The entry to the accepted state is a violent operation in which the device will reject MMIO requests that are not encrypted, and the device enters a new IOMMU protection domain to allow it to access addresses that were previously off-limits.Trying to mix/match "acceptance" with "authorized" is going to be a nightmare, what's the combination that can happen here over time?
Quoted 6 lines
Quoted 3 lines
I do think Linux needs to mix/match these concepts. "Authorization" is a kernel policy to operate a device at all. "Acceptance" is a mechanism to operate a device within a hardware TCB boundary.
I'm not sure about these words either, I would revise your table to be more OS centric, the device can be in one of four security levels:
I like this framing.
Quoted 3 lines
0 Blocked and disabled
The device cannot attack the system, enforced by the OS not loading a driver or mapping the MMIO and IOMMU fully blocking everything from it.
In terms of details I am trying to think through whether the device actually changes its ->trust level in reaction to a driver attaching, or whether the block and disabled state is implicit in not being driver bound.
It does strike me that this value could be used to convey whether a given arch's IOMMU driver indeed arranges for devices to be IOMMU blocked while driver detached. In that case you could see, "oh, devices are not DMA blocked by default" as we talked about in the ATS-always-on thread [1].
[1]: http://lore.kernel.org/[email protected]
Quoted 5 lines
1 In use, attacks from a hostile device are possible
A driver can operate the device and is expected to defend against attacks from the device itself. The IOMMU restricts the device to only access driver approved data (no ATS, DMA strict only, CC shared only, interrupt remapping security, bounce partial DMA mappings, etc)
This is a better way to convey the current "force_swiotlb" settings that TVMs deploy in their arch code.
Quoted 8 lines
2 In use, no attacks from the device
The device does what the driver says and is not hostile. The driver does not have to defend itself, the IOMMU can run in faster & lower security modes (ATS on, DMA-FQ, Identity, still CC shared only)
* Basically our default security level today3 In use, no attacks, and access to CC private memory
Like #2 and now the IOMMU allows access to CC private memory too.
I am assuming that each bus implementation may have a different way to get the device to the various trust levels.
For example, the uAPI for PCI TDISP requires associating a device with a TSM and asking the TSM to push the device to trust level 3. Another bus like thunderbolt may want to imply that "authorized" that uses challenge response (tb_domain_challenge_switch_key) enables trust level 2, but otherwise only enables trust level 1.
Quoted 7 lines
[*] I'm inclunding all attacks with "hostile device", including MIM on the PCIe link, compromised/fake device, attacks from a VMM through a virtual device, etc.
From a CC VM perspective 0 is at boot, 1 is an out of TCB device, 2 doesn't exist (without TDISP there is no way to keep the hypervisor from attacking?),
Yes, no mitigations against spoofing the device interface without TDISP. However, I would also assume that level 2 is the ATS-on trust level outside of TDISP cases.
Quoted 4 lines
and 3 is a full accepted TDSIP device.
#2 can happen in bare metal where a OS may activate link encryption and attest the device, but doesn't have CC private/shared memory.
Bare metal would still need to figure out how to send T=1 MMIO cycles and check with some boot attestation that it can trust its MMIO mappings are indeed targeting the device. So let's say trust level 2 is everything but private MMIO and private DMA.
Quoted 4 lines
From a uAPI perspective I'm not sold on having two bools, I think a level string would be more flexible. TSM and CC properties are orthogonal, except you can't select #3 without the TSM saying it is in RUN.
Perhaps the concern is less 2 bools in the uAPI and more the concern that 'struct pci_dev::untrusted', 'struct tb_switch::authorized', 'struct usb_dev::authorized' and this new 'struct device_private::cc_accepted' are getting convoluted.
Quoted 2 lines
Internally we'd probably turn that
dev->trustedthing into an enum and teach the iommu layer to treat it more dynamically.
I will take a stab at some patches in this direction and at least demonstrate how 'struct pci_dev::untrusted' can be merged with what CC wants to add on top.
Jason Gunthorpe
maintainerRe: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 8 lines · Dan Williams
Quoted 3 lines
0 Blocked and disabled
The device cannot attack the system, enforced by the OS not loading a driver or mapping the MMIO and IOMMU fully blocking everything from it.In terms of details I am trying to think through whether the device actually changes its ->trust level in reaction to a driver attaching, or whether the block and disabled state is implicit in not being driver bound.
I am thinking of it as an independent property. When the device is first discovered it gets a level by default, userspace can change the level but only when not bound. The level restricts what the kernel will do with the device, 0 would mean "do not allow a driver to bind"
Quoted 8 lines
Quoted 5 lines
1 In use, attacks from a hostile device are possible
A driver can operate the device and is expected to defend against attacks from the device itself. The IOMMU restricts the device to only access driver approved data (no ATS, DMA strict only, CC shared only, interrupt remapping security, bounce partial DMA mappings, etc)This is a better way to convey the current "
force_swiotlb" settings that TVMs deploy in their arch code.
SWIOTLB that is needed to make the DMA API work because the device cannot reach CC private memory is orthogonal - the TDISP state (or lack of) should directly drive that in the DMA API.
The DMA API just wants a flag in the struct device that says if the device can access encrypted memory or only decrypted.
Quoted 2 lines
I am assuming that each bus implementation may have a different way to get the device to the various trust levels.
I was actually thinking no, it is just a generic orthogonal driver core property.
Quoted 2 lines
For example, the uAPI for PCI TDISP requires associating a device with a TSM and asking the TSM to push the device to trust level 3.
The other way, you can't get to level 3 unless the TSM subsystem ACK's it. So TSM independently does its bit then userspace can set the level to 3.
If it sets RUN and 2 that should work and have some kind of meaning, just not be super useful.
Quoted 3 lines
Another bus like thunderbolt may want to imply that "authorized" that uses challenge response (
tb_domain_challenge_switch_key) enables trust level 2, but otherwise only enables trust level 1.
For thunderbolt/hot plug I imagine the kernel would default all devices to level 0. Userspace would do its thing, using whatever other uAPIs, and then set the level to 1 or 2. Then the driver starts.
This way nothing is coupled and the kernel can offer all kinds of different uAPI for device verification. Userspaces picks the appropriate one and acks it with the level change.
Quoted 3 lines
Yes, no mitigations against spoofing the device interface without TDISP. However, I would also assume that level 2 is the ATS-on trust level outside of TDISP cases.
Yes, level 2 would be the break where the device is required to not do wild PCIe packets to maintain kernel integrity.
Quoted 7 lines
Quoted 2 lines
#2 can happen in bare metal where a OS may activate link encryption and attest the device, but doesn't have CC private/shared memory.
Bare metal would still need to figure out how to send T=1 MMIO cycles and check with some boot attestation that it can trust its MMIO mappings are indeed targeting the device. So let's say trust level 2 is everything but private MMIO and private DMA.
bare metal has no T=1 and no "private" at all. It just sets up link encryption, excludes a MIM, attests the peer, then opens the iommu.
Jason
Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 10 lines · Jason Gunthorpe
Quoted 8 lines · Dan Williams
Quoted 3 lines
0 Blocked and disabled
The device cannot attack the system, enforced by the OS not loading a driver or mapping the MMIO and IOMMU fully blocking everything from it.In terms of details I am trying to think through whether the device actually changes its ->trust level in reaction to a driver attaching, or whether the block and disabled state is implicit in not being driver bound.
Quoted 4 lines
I am thinking of it as an independent property. When the device is first discovered it gets a level by default, userspace can change the level but only when not bound. The level restricts what the kernel will do with the device, 0 would mean "do not allow a driver to bind"
The problem is that for all the buses that do not currently have a "device authorization" concept only userspace can decide that a device should skip bind by default. For that, I propose module autoprobe policy [1]. Not yet convinced the kernel needs its own per-device "no bind" policy.
However, I do think userspace would like to know if the IOMMU subsystem has blocked device DMA while unbound.
[1]: http://lore.kernel.org/[email protected]
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
1 In use, attacks from a hostile device are possible
A driver can operate the device and is expected to defend against attacks from the device itself. The IOMMU restricts the device to only access driver approved data (no ATS, DMA strict only, CC shared only, interrupt remapping security, bounce partial DMA mappings, etc)This is a better way to convey the current "
force_swiotlb" settings that TVMs deploy in their arch code.
Quoted 6 lines
SWIOTLB that is needed to make the DMA API work because the device cannot reach CC private memory is orthogonal - the TDISP state (or lack of) should directly drive that in the DMA API.
The DMA API just wants a flag in the struct device that says if the device can access encrypted memory or only decrypted.
You mean separate "trusted to access private" and "currently enabled to access private" properties? I am trying to think of a situation where "dev->trust >= 3" and a flag saying "disable bouncing for encrypted memory" would ever disagree.
Quoted 5 lines
Quoted 2 lines
I am assuming that each bus implementation may have a different way to get the device to the various trust levels.
I was actually thinking no, it is just a generic orthogonal driver core property.
Property? Agreed. uAPI? Not so sure...
Quoted 6 lines
Quoted 2 lines
For example, the uAPI for PCI TDISP requires associating a device with a TSM and asking the TSM to push the device to trust level 3.
The other way, you can't get to level 3 unless the TSM subsystem ACK's it. So TSM independently does its bit then userspace can set the level to 3.
That bit though has lock-to-run consistency expectations. So if the kernel does not yet fully trust the device by time the relying party is satisfied, and the uAPI to transition the device into the TCB (level 3) is driver-core generic it raises TOCTOU issues in my mind. The driver-core would need to ask the bus "user now trusts this device, do you?".
Aneesh and I are currently debating on Discord whether the kernel needs to protect against guest userspace confusing itself. Part of me says no, especially with sysfs, if multiple threads are racing "unlock, update/re-measure, lock, accept", then userspace gets to keep the pieces.
However, to Aneesh's point we could protect against that with a transactional uAPI like netlink that can express "trust if and only if the device has not been relocked before final accept" by passing a cookie obtained at lock to accept. That would be awkward to coordinate with driver-core generic uAPI for trust.
Quoted 7 lines
If it sets RUN and 2 that should work and have some kind of meaning, just not be super useful.
Quoted 3 lines
Another bus like thunderbolt may want to imply that "authorized" that uses challenge response (
tb_domain_challenge_switch_key) enables trust level 2, but otherwise only enables trust level 1.
Quoted 7 lines
For thunderbolt/hot plug I imagine the kernel would default all devices to level 0. Userspace would do its thing, using whatever other uAPIs, and then set the level to 1 or 2. Then the driver starts.
This way nothing is coupled and the kernel can offer all kinds of different uAPI for device verification. Userspaces picks the appropriate one and acks it with the level change.
Thunderbolt already has authorized uAPI. I expect adding dev->trust support to thunderbolt is more related to ATS privilege and private memory privilege.
Quoted 10 lines
Quoted 3 lines
Yes, no mitigations against spoofing the device interface without TDISP. However, I would also assume that level 2 is the ATS-on trust level outside of TDISP cases.
Yes, level 2 would be the break where the device is required to not do wild PCIe packets to maintain kernel integrity.
Quoted 3 lines
Quoted 2 lines
#2 can happen in bare metal where a OS may activate link encryption and attest the device, but doesn't have CC private/shared memory.
Quoted 7 lines
Quoted 4 lines
Bare metal would still need to figure out how to send T=1 MMIO cycles and check with some boot attestation that it can trust its MMIO mappings are indeed targeting the device. So let's say trust level 2 is everything but private MMIO and private DMA.
bare metal has no T=1 and no "private" at all. It just sets up link encryption, excludes a MIM, attests the peer, then opens the iommu.
Ok, we are on the same page as to what a theoretical level 2 would mean.
Jason Gunthorpe
maintainerRe: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 5 lines · Dan Williams
The problem is that for all the buses that do not currently have a "device authorization" concept only userspace can decide that a device should skip bind by default. For that, I propose module autoprobe policy [1]. Not yet convinced the kernel needs its own per-device "no bind" policy.
I think it is just part of the broader definition of the level that extends into the iommu and so on. It makes sense to have this kind of no-binding security level, IMHO.
Quoted 7 lines
Quoted 2 lines
The DMA API just wants a flag in the struct device that says if the device can access encrypted memory or only decrypted.
You mean separate "trusted to access private" and "currently enabled to access private" properties? I am trying to think of a situation where "
dev->trust>= 3" and a flag saying "disable bouncing for encrypted memory" would ever disagree.
I'm steering the trust level toward more of an acceptance criteria. If the trust level is you have access to private memory but the device can't actually do that then fail the trust level change.
Same for the reverse, if the trust level says no private memory and the device is T=1 then fail the trust level change.
Quoted 6 lines
That bit though has lock-to-run consistency expectations. So if the kernel does not yet fully trust the device by time the relying party is satisfied, and the uAPI to transition the device into the TCB (level 3) is driver-core generic it raises TOCTOU issues in my mind. The driver-core would need to ask the bus "user now trusts this device, do you?".
Huh? No, there is no concept of trust in the kernel. The userspace setting level 3 is "I now ack that this device is trusted", there is no further trust cross check. If TSM side says it is in RUN/T=1 then we are done.
If we fall out of RUN then the level auto-resets back to 0 and userspace has to go around and fix it again. (ignoring driver RAS)
Quoted 2 lines
Aneesh and I are currently debating on Discord whether the kernel needs to protect against guest userspace confusing itself.
Userspace that controls acceptance must be part of the TCB or the whole model is fully broken. If your guest userspace is so security broken it can accept devices it doesn't mean to then just forget it.
Quoted 5 lines
However, to Aneesh's point we could protect against that with a transactional uAPI like netlink that can express "trust if and only if the device has not been relocked before final accept" by passing a cookie obtained at lock to accept. That would be awkward to coordinate with driver-core generic uAPI for trust.
You could, but why make it so complicated? The whole LOCKED/RUN thing is already supposed to deal with TOCTOU, doesn't it? The CSP cannot trick a device to fall out of LOCKED an the re-enter LOCKED without the VM knowing.
The VM attacking itself on something as security critical as device accepance can't be in scope :\
Quoted 7 lines
Quoted 3 lines
This way nothing is coupled and the kernel can offer all kinds of different uAPI for device verification. Userspaces picks the appropriate one and acks it with the level change.
Thunderbolt already has authorized uAPI. I expect adding
dev->trustsupport to thunderbolt is more related to ATS privilege and private memory privilege.
It brings it into the whole 'measure the device and then decide what to do with it' framework. The trust level is still the generic ack that the device is allowed the participate in the system with whatever level of security.
Jason
Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 11 lines · Jason Gunthorpe
Quoted 5 lines · Dan Williams
The problem is that for all the buses that do not currently have a "device authorization" concept only userspace can decide that a device should skip bind by default. For that, I propose module autoprobe policy [1]. Not yet convinced the kernel needs its own per-device "no bind" policy.
I think it is just part of the broader definition of the level that extends into the iommu and so on. It makes sense to have this kind of no-binding security level, IMHO.
Easy to include for completeness.
In a CC VM userspace can set "$module autoprobe=0" to get a control point to set @trust to zero, but @trust otherwise defaults to 1. This allows for userspace policy to generically distrust devices, but without needing to build a new mechanism to specify which devices start life at trust == 0 (i.e. "device filter" proposal previously NAK'd).
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
The DMA API just wants a flag in the struct device that says if the device can access encrypted memory or only decrypted.
You mean separate "trusted to access private" and "currently enabled to access private" properties? I am trying to think of a situation where "
dev->trust>= 3" and a flag saying "disable bouncing for encrypted memory" would ever disagree.
Quoted 6 lines
I'm steering the trust level toward more of an acceptance criteria. If the trust level is you have access to private memory but the device can't actually do that then fail the trust level change.
Same for the reverse, if the trust level says no private memory and the device is T=1 then fail the trust level change.
Ok, so the uapi for PCI/TDISP would be:
echo $tsm > $pdev/tsm/lock
<gather evidence, validate with relying party>
echo 3 > $pdev/trust
...where that @trust attribute is a generic device semantic, but in the case of PCI device connected to a given TSM it invokes the TSM hypercall to transition the device to the RUN state and the TSM local call to unblock DMA to private memory.
So, userspace can generically understand what privileges come with which trust levels, but the mechanism to get those privileges remains bus specific.
Quoted 11 lines
Quoted 6 lines
That bit though has lock-to-run consistency expectations. So if the kernel does not yet fully trust the device by time the relying party is satisfied, and the uAPI to transition the device into the TCB (level 3) is driver-core generic it raises TOCTOU issues in my mind. The driver-core would need to ask the bus "user now trusts this device, do you?".
Huh? No, there is no concept of trust in the kernel. The userspace setting level 3 is "I now ack that this device is trusted", there is no further trust cross check. If TSM side says it is in RUN/T=1 then we are done.
Ok, so maybe I misunderstood your point. Per the above, if the trust setting is what kicks the bus to finalize T=1 then it makes sense. If that kick fails, the user's trust setting request fails.
What I expect is unwanted / surprising is device has already been transitioned to T=1 state ahead of the trust setting, it is a synchronous mechanism.
Quoted 2 lines
If we fall out of RUN then the level auto-resets back to 0 and userspace has to go around and fix it again. (ignoring driver RAS)
Yes, at the moment that the bus detects that an event like SPDM session loss, IDE link loss, TDISP ERROR state entry has occurred it can downgrade trust and notify. That notification fits well with netlink because all of those events are downstream of evidence validation.
Quoted 6 lines
Quoted 2 lines
Aneesh and I are currently debating on Discord whether the kernel needs to protect against guest userspace confusing itself.
Userspace that controls acceptance must be part of the TCB or the whole model is fully broken. If your guest userspace is so security broken it can accept devices it doesn't mean to then just forget it.
Agree, it is a non-problem. If guest userspace confuses itself by racing sysfs operations then the relying party should not trust that userspace.
Quoted 8 lines
Quoted 5 lines
However, to Aneesh's point we could protect against that with a transactional uAPI like netlink that can express "trust if and only if the device has not been relocked before final accept" by passing a cookie obtained at lock to accept. That would be awkward to coordinate with driver-core generic uAPI for trust.
You could, but why make it so complicated? The whole LOCKED/RUN thing is already supposed to deal with TOCTOU, doesn't it?
Right, the threat vector is the guest accepting something it has had no chance to validate. Guest userspace confusion is not that. Guest userspace asking the device to be re-locked in a way that confuses an ongoing evidence validation sequence in another thread is a "you get to keep the pieces" event.
Quoted 5 lines
The CSP cannot trick a device to fall out of LOCKED an the re-enter LOCKED without the VM knowing.
The VM attacking itself on something as security critical as device accepance can't be in scope :\
The complication vs benefit tradeoff is indeed not mathing, but wanted to do justice to Aneesh's proposal and the suitability of the sysfs uapi.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
This way nothing is coupled and the kernel can offer all kinds of different uAPI for device verification. Userspaces picks the appropriate one and acks it with the level change.
Quoted 8 lines
Quoted 3 lines
Thunderbolt already has authorized uAPI. I expect adding
dev->trustsupport to thunderbolt is more related to ATS privilege and private memory privilege.It brings it into the whole 'measure the device and then decide what to do with it' framework. The trust level is still the generic ack that the device is allowed the participate in the system with whatever level of security.
Some thought experiments to confirm alignment...
'Generic ack' is a synchronous mechanism for the bus to evaluate. So if @trust appears for any device, and by consequence alongside @authorized for a thunderbolt device, it should be the case that these operations are equivalent:
# echo 1 > $dev/trust # echo 1 > $dev/authorized
...and the result is cross-reflected for comptability:
# echo 1 > $dev/trust # cat $dev/authorized 1
Consequently this:
# echo 2 > $dev/trust
...would be equivalent to authorizing the device and unblocking ATS (if such a thing existed).
For bare metal PCI device security the TSM 'connection' needs to be established in order to enable device evidence collection.
echo $tsm > $pdev/tsm/connect
<validate device evidence>
echo 2 > $pdev/trust
Now, I question whether 5 trust levels instead of 4. This would be to explicitly only trust devices where the TSM has established physical link encryption, or the TSM has asserted that the link is protected by other means. So the trust levels are:
0 disconnected: bus does not attach drivers
1 limited: core code deploys hostile device mitigations like disable
ATS, CC force shared memory bouncing.
2 DMA: full DMA access, driver responsible for protecting against
adverarial devices.
3 Link: mitigations against physical integrity and snooping attacks
deployed
4 TCB: full device interface validation via a protocol like TDISP,
CC private memory access granted.
Where the native Rust library based SPDM driver only offers trust level 2, bare metal TSMs can support trust level 3, and the TSM interfaces in CC VMs can support trust level 4. I could see squashing 2 and 3 and making it a documentation problem to understand the capabilities of the various TSM drivers.
Jason Gunthorpe
maintainerRe: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 13 lines · Dan Williams
Quoted 11 lines · Jason Gunthorpe
Quoted 5 lines · Dan Williams
The problem is that for all the buses that do not currently have a "device authorization" concept only userspace can decide that a device should skip bind by default. For that, I propose module autoprobe policy [1]. Not yet convinced the kernel needs its own per-device "no bind" policy.
I think it is just part of the broader definition of the level that extends into the iommu and so on. It makes sense to have this kind of no-binding security level, IMHO.
Quoted 7 lines
Easy to include for completeness.
In a CC VM userspace can set "$module autoprobe=0" to get a control point to set @trust to zero, but @trust otherwise defaults to 1. This allows for userspace policy to generically distrust devices, but without needing to build a new mechanism to specify which devices start life at trust == 0 (i.e. "device filter" proposal previously NAK'd).
I feel like starting with trust=0 is much cleaner than using autoprobe. Especially since it would be nice that when you do ultimately set trust!=0 then you do want the kernel to do the normal autoprobe flow.
Double so because I would like the iommu drivers to respond to trust 0 by fully blocking the device 100% of the time without holes, so to make that work I would like to see the struct device report trust 0 the moment the iommu framework attaches the iommu.
How you decide the starting trust value for device during system boot is definately something we need to discuss properly..
I liked your idea of using built in driver match, so if there is a simple command line paramater that says 'only built in is trusted' then we'd default all devices to untrusted and during device probe check if any built in driver is matching and auto-set trust to X based on the commandline parameter.
With the idea that only devices required to get to the initrd are built in. Then the initrd userspace has the policy to bring more devices into trusted!=0 to get to the root file system, then the rootfs has more policy for further devices, and so on.
Probably this would ultimately escalate into annotations in the modinfo about default policies for various drivers.
A kernel default policy of trusting everything without a "trust ops" (see below) may also be quite reasonable, however boot ordering the trust ops might be really tricky...
Quoted 17 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
The DMA API just wants a flag in the struct device that says if the device can access encrypted memory or only decrypted.
You mean separate "trusted to access private" and "currently enabled to access private" properties? I am trying to think of a situation where "
dev->trust>= 3" and a flag saying "disable bouncing for encrypted memory" would ever disagree.Quoted 6 lines
I'm steering the trust level toward more of an acceptance criteria. If the trust level is you have access to private memory but the device can't actually do that then fail the trust level change.
Same for the reverse, if the trust level says no private memory and the device is T=1 then fail the trust level change.
Ok, so the uapi for PCI/TDISP would be:
Quoted 8 lines
echo $tsm > $pdev/tsm/lock
<gather evidence, validate with relying party>
echo 3 > $pdev/trust...where that @trust attribute is a generic device semantic, but in the case of PCI device connected to a given TSM it invokes the TSM hypercall to transition the device to the RUN state and the TSM local call to unblock DMA to private memory.
Maybe, but I was thinking the transition through run/locked would be done through TSM uAPIs too. trust setting in the kernel just confirms the device is in the right state.
But I haven't thought of a reason why the final switch to RUN couldn't happen like this either.
Quoted 3 lines
So, userspace can generically understand what privileges come with which trust levels, but the mechanism to get those privileges remains bus specific.
Yes
Quoted 7 lines
Quoted 2 lines
If we fall out of RUN then the level auto-resets back to 0 and userspace has to go around and fix it again. (ignoring driver RAS)
Yes, at the moment that the bus detects that an event like SPDM session loss, IDE link loss, TDISP ERROR state entry has occurred it can downgrade trust and notify. That notification fits well with netlink because all of those events are downstream of evidence validation.
Which is also why it would be nice to be consistent and rely on trust=0 to isolate the device in all cases, not a mixture of autoprobe.
Quoted 3 lines
The complication vs benefit tradeoff is indeed not mathing, but wanted to do justice to Aneesh's proposal and the suitability of the sysfs uapi.
I think if you want something like this then it is better to target the root - remove the ability for concurrent userspace to wrongly operate the TSM entirely. Ie use a cdev, make it so going to LOCKED isolates access to only this cdev fd and require only this cdev fd to go to RUN. Then these kinds of bugs don't exist.
Quoted 21 lines
Quoted 4 lines
It brings it into the whole 'measure the device and then decide what to do with it' framework. The trust level is still the generic ack that the device is allowed the participate in the system with whatever level of security.
Some thought experiments to confirm alignment...
'Generic ack' is a synchronous mechanism for the bus to evaluate. So if @trust appears for any device, and by consequence alongside @authorized for a thunderbolt device, it should be the case that these operations are equivalent:
# echo 1 > $dev/trust # echo 1 > $dev/authorized...and the result is cross-reflected for comptability:
# echo 1 > $dev/trust # cat $dev/authorized 1
Quoted 6 lines
Consequently this:
# echo 2 > $dev/trust...would be equivalent to authorizing the device and unblocking ATS (if such a thing existed).
Yes, I don't know anything about thunderbolt, but this seems reasonable. You could also do as you suggested for TDISP that trust!=0 auto-authorizes.
Basically the 'trust' generic framework sits on top of some "trust ops" that will be provided by the security module that is affiliated with the struct device (ie thunderbolt, TSM TDISP, TSM Link IDE, etc, etc)
Then it becomes a general synchronization point where on one side the "tust ops" can ack that the level is acceptable and consistent with the system when on the other side generic compoments like IOMMU, driver binding, etc can respond to it and change their behavior.
Quoted 11 lines
For bare metal PCI device security the TSM 'connection' needs to be established in order to enable device evidence collection.
echo $tsm > $pdev/tsm/connect
<validate device evidence>
echo 2 > $pdev/trustNow, I question whether 5 trust levels instead of 4. This would be to explicitly only trust devices where the TSM has established physical link encryption, or the TSM has asserted that the link is protected by other means. So the trust levels are:
I probably wouldn't use an int for the uAPI, but yes picking the initial levels is important. As above since this is a clearing point between two different worlds it needs to be defined in some way both sides can understand what it means for them.
Quoted 9 lines
0 disconnected: bus does not attach drivers
1 limited: core code deploys hostile device mitigations like disable
ATS, CC force shared memory bouncing.
2 DMA: full DMA access, driver responsible for protecting against
adverarial devices.
3 Link: mitigations against physical integrity and snooping attacks
deployed
4 TCB: full device interface validation via a protocol like TDISP,
CC private memory access granted.
This seems reasonable to me, the 3/4 distinction is not meaningful for the iommu&dev side, but it does provide a good check point for the "trust ops". If userspace ack's that it expects physical security and the kernel says it isn't physically secure (or becomes insecure later) then it should fail.
Quoted 3 lines
Where the native Rust library based SPDM driver only offers trust level 2, bare metal TSMs can support trust level 3, and the TSM interfaces in CC VMs can support trust level 4.
I'm not sure that the SPDM driver even provides a "trust ops" right? I would guess that 0/1/2 are simply built in always available if trust ops are NULL and 3/4 require positive reply from the ops to accept it.
So #3 needs a "trust ops" linked to enabling link IDE.. If this is done in-kernel the link IDE module is providing the trust ops and just using SPDM as a library to establish the link IDE keys?
Jason
Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Jason Gunthorpe wrote:
[..]
Quoted 10 lines
I feel like starting with trust=0 is much cleaner than using autoprobe. Especially since it would be nice that when you do ultimately set trust!=0 then you do want the kernel to do the normal autoprobe flow.
Double so because I would like the iommu drivers to respond to trust 0 by fully blocking the device 100% of the time without holes, so to make that work I would like to see the struct device report trust 0 the moment the iommu framework attaches the iommu.
Quoted 8 lines
How you decide the starting trust value for device during system boot is definately something we need to discuss properly..
I liked your idea of using built in driver match, so if there is a simple command line paramater that says 'only built in is trusted' then we'd default all devices to untrusted and during device probe check if any built in driver is matching and auto-set trust to X based on the commandline parameter.
I do agree that forcing trust=0 at the beginning of time is attractive and theoretically clean. I am concerned about subsystems that are not prepared for driver attach failures. For example, I would not expect to need to set trust for auxiliary bus devices if the host device is trusted.
However, the work to set module autoprobe policy is on the same order as adding a module scoped trust policy.
So something like "modprobe $module trust=X" automatically tries to set the device trust level on attach to any drivers in that module. That could allow a semantic of "attach iff device is able to go to level 4".
Quoted 7 lines
With the idea that only devices required to get to the initrd are built in. Then the initrd userspace has the policy to bring more devices into trusted!=0 to get to the root file system, then the rootfs has more policy for further devices, and so on.
Probably this would ultimately escalate into annotations in the modinfo about default policies for various drivers.
Yes, escalate over time for subsystems to say "devices I create are trusted, the responsibility to manage trust lies with clients of my APIs".
Quoted 3 lines
A kernel default policy of trusting everything without a "trust ops" (see below) may also be quite reasonable, however boot ordering the trust ops might be really tricky...
Given the optionality in selecting a trust ops provider I think it gets extra messy quickly. Let me see how far I can get with built-in auto trust + module trust policy.
Quoted 26 lines
Quoted 17 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
The DMA API just wants a flag in the struct device that says if the device can access encrypted memory or only decrypted.
You mean separate "trusted to access private" and "currently enabled to access private" properties? I am trying to think of a situation where "
dev->trust>= 3" and a flag saying "disable bouncing for encrypted memory" would ever disagree.Quoted 6 lines
I'm steering the trust level toward more of an acceptance criteria. If the trust level is you have access to private memory but the device can't actually do that then fail the trust level change.
Same for the reverse, if the trust level says no private memory and the device is T=1 then fail the trust level change.
Ok, so the uapi for PCI/TDISP would be:
Quoted 8 lines
echo $tsm > $pdev/tsm/lock
<gather evidence, validate with relying party>
echo 3 > $pdev/trust...where that @trust attribute is a generic device semantic, but in the case of PCI device connected to a given TSM it invokes the TSM hypercall to transition the device to the RUN state and the TSM local call to unblock DMA to private memory.
Quoted 6 lines
Maybe, but I was thinking the transition through run/locked would be done through TSM uAPIs too. trust setting in the kernel just confirms the device is in the right state.
But I haven't thought of a reason why the final switch to RUN couldn't happen like this either.
Right, the potential to see in-between states concerns me because TSM uAPIs would have fully enabled the device to wreak havoc, meanwhile dev->trust is still showing the device at some lower level of trust. So I think trust modification needs to be synchronous with privileges granted/revoked.
[..]
Quoted 9 lines
Quoted 3 lines
The complication vs benefit tradeoff is indeed not mathing, but wanted to do justice to Aneesh's proposal and the suitability of the sysfs uapi.
I think if you want something like this then it is better to target the root - remove the ability for concurrent userspace to wrongly operate the TSM entirely. Ie use a cdev, make it so going to LOCKED isolates access to only this cdev fd and require only this cdev fd to go to RUN. Then these kinds of bugs don't exist.
The netlink evidence proposal can handle this, it just needs a 'validate' command. 'Validate' records a device evidence generation number. Require a stable generation number between entering LOCK and the trust=4 event transitioning the device to RUN.
Yes, not as safe as fd-private LOCK-to-RUN, but I like semantic of "modprobe $module trust=4" to say "transition to RUN and attach to all validated devices".
[..]
Quoted 9 lines
Basically the 'trust' generic framework sits on top of some "trust ops" that will be provided by the security module that is affiliated with the struct device (ie thunderbolt, TSM TDISP, TSM Link IDE, etc, etc)
Then it becomes a general synchronization point where on one side the "tust ops" can ack that the level is acceptable and consistent with the system when on the other side generic compoments like IOMMU, driver binding, etc can respond to it and change their behavior.
Something like this, yes.
Quoted 7 lines
Quoted 7 lines
For bare metal PCI device security the TSM 'connection' needs to be established in order to enable device evidence collection.
echo $tsm > $pdev/tsm/connect
<validate device evidence>
echo 2 > $pdev/trust
Quoted 6 lines
Quoted 4 lines
Now, I question whether 5 trust levels instead of 4. This would be to explicitly only trust devices where the TSM has established physical link encryption, or the TSM has asserted that the link is protected by other means. So the trust levels are:
I probably wouldn't use an int for the uAPI,
Yes, but an int for now saves the bikeshed of the level names till a bit later.
Quoted 20 lines
...but yes picking the initial levels is important. As above since this is a clearing point between two different worlds it needs to be defined in some way both sides can understand what it means for them.
Quoted 9 lines
0 disconnected: bus does not attach drivers
1 limited: core code deploys hostile device mitigations like disable
ATS, CC force shared memory bouncing.
2 DMA: full DMA access, driver responsible for protecting against
adverarial devices.
3 Link: mitigations against physical integrity and snooping attacks
deployed
4 TCB: full device interface validation via a protocol like TDISP,
CC private memory access granted.This seems reasonable to me, the 3/4 distinction is not meaningful for the iommu&dev side, but it does provide a good check point for the "trust ops". If userspace ack's that it expects physical security and the kernel says it isn't physically secure (or becomes insecure later) then it should fail.
Quoted 7 lines
Quoted 3 lines
Where the native Rust library based SPDM driver only offers trust level 2, bare metal TSMs can support trust level 3, and the TSM interfaces in CC VMs can support trust level 4.
I'm not sure that the SPDM driver even provides a "trust ops" right? I would guess that 0/1/2 are simply built in always available if trust ops are NULL and 3/4 require positive reply from the ops to accept it.
Right. Even though the SPDM driver allows device evidence to be collected there is no event like LOCK-to-RUN that expects validated evidence as a precondition.
Quoted 3 lines
So #3 needs a "trust ops" linked to enabling link IDE.. If this is done in-kernel the link IDE module is providing the trust ops and just using SPDM as a library to establish the link IDE keys?
Yes, but note to date only Intel platforms allow for IDE establishment without talking to the platform TSM.
Jason Gunthorpe
maintainerRe: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 21 lines · Dan Williams
Jason Gunthorpe wrote:
[..]Quoted 10 lines
I feel like starting with trust=0 is much cleaner than using autoprobe. Especially since it would be nice that when you do ultimately set trust!=0 then you do want the kernel to do the normal autoprobe flow.
Double so because I would like the iommu drivers to respond to trust 0 by fully blocking the device 100% of the time without holes, so to make that work I would like to see the struct device report trust 0 the moment the iommu framework attaches the iommu.
Quoted 8 lines
How you decide the starting trust value for device during system boot is definately something we need to discuss properly..
I liked your idea of using built in driver match, so if there is a simple command line paramater that says 'only built in is trusted' then we'd default all devices to untrusted and during device probe check if any built in driver is matching and auto-set trust to X based on the commandline parameter.
Quoted 5 lines
I do agree that forcing trust=0 at the beginning of time is attractive and theoretically clean. I am concerned about subsystems that are not prepared for driver attach failures. For example, I would not expect to need to set trust for auxiliary bus devices if the host device is trusted.
Yeah, IDK here either. Maybe it is some per-bus opt int.
I think the most important thing is we get a clean story for devices to be isolated by the iommu until user space ack's them.
Quoted 5 lines
Right, the potential to see in-between states concerns me because TSM uAPIs would have fully enabled the device to wreak havoc, meanwhile
dev->trustis still showing the device at some lower level of trust. So I think trust modification needs to be synchronous with privileges granted/revoked.
If an iommu is present then the device will still be blocked even though it is in RUN, I'm not sure this synchronicity is so important.
Jason
Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Jason Gunthorpe wrote:
[..]
Quoted 8 lines
Quoted 5 lines
Right, the potential to see in-between states concerns me because TSM uAPIs would have fully enabled the device to wreak havoc, meanwhile
dev->trustis still showing the device at some lower level of trust. So I think trust modification needs to be synchronous with privileges granted/revoked.If an iommu is present then the device will still be blocked even though it is in RUN, I'm not sure this synchronicity is so important.
Oh, maybe we are just quibbling about where the mechanism lives. The "unblock DMA" step in current preliminary patches is currently behind the "struct pci_tsm_ops::accept()" op which also handles transitioning the device to RUN / T=1. It is a bus callback.
However, if the IOMMU layer is enlightened to block/unblock DMA on trust setting then the TDISP "unblock DMA" step can be factored out of this bus callback and into the IOMMU trust responder.
So device could enter T=1 way in advance of the "unblock DMA" event.
I assume this would also expect that encrypted MMIO mappings are also not established while trust is less than "TCB"? That would require some additional enabling to catch attempts to establish an encrypted mapping that the hardware is prepared for, but dev->trust is not, all without needing to modify the driver to worry about this difference. Drivers would just see ioremap() failure in this case.
A bit more work, but yes, that is a cleaner separation of concerns.
Jason Gunthorpe
maintainerRe: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 11 lines · Dan Williams
Jason Gunthorpe wrote:
[..]Quoted 8 lines
Quoted 5 lines
Right, the potential to see in-between states concerns me because TSM uAPIs would have fully enabled the device to wreak havoc, meanwhile
dev->trustis still showing the device at some lower level of trust. So I think trust modification needs to be synchronous with privileges granted/revoked.If an iommu is present then the device will still be blocked even though it is in RUN, I'm not sure this synchronicity is so important.
Quoted 8 lines
Oh, maybe we are just quibbling about where the mechanism lives. The "unblock DMA" step in current preliminary patches is currently behind the "struct
pci_tsm_ops::accept()" op which also handles transitioning the device to RUN / T=1. It is a bus callback.However, if the IOMMU layer is enlightened to block/unblock DMA on trust setting then the TDISP "unblock DMA" step can be factored out of this bus callback and into the IOMMU trust responder.
Yes, I would prefer this because it makes the whole IOMMU mechanism entirely general and not tied to TDISP - which I think is sort of what Greg is pushing on too.
Quoted 6 lines
I assume this would also expect that encrypted MMIO mappings are also not established while trust is less than "TCB"? That would require some additional enabling to catch attempts to establish an encrypted mapping that the hardware is prepared for, but
dev->trustis not, all without needing to modify the driver to worry about this difference. Drivers would just seeioremap()failure in this case.
Hmm.. I don't know if this matters. Once we decide to use the device the MMIO should be mapped in the correct way, whatever that is.
If we decide to eventually allow a lower trust while T=1 then that should be taken to mean the user wants all the features protecting the communication channel but also all the IOMMU features restricting what memory the device can access.
Remember there are two parallel things here, one is T=1 which is designed to protect against hypervisor and physical attacks, the other is the trust level and iommu which would be able to protect against attacks from an attested device itself.
Even if you are in a T=1 environment you may still decide you don't really trust the device firmware that much and would prefer to have it more restricted.
For example, if you have a system with a NVMe drive then all the data on the drive is probably still encrypted and has be CPU-decrypted before it can be used. It would be reasonable to run in T=1 and attest the drive to limit attack surface but also use the IOMMU to limit NVMe access to only the memory used to bounce to the CPU decryption as an additional fortification.
This is why I am tending to prefer that the kernel's view of trust level and the physical HW capability are somewhat orthogonal things. Even if the HW has high security the user may still prefer that the kernel distrust.
Jason
Greg KH
maintainerRe: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 17 lines · Jason Gunthorpe
Quoted 16 lines · Dan Williams
Jason Gunthorpe wrote:
[..]Quoted 8 lines
Quoted 5 lines
Right, the potential to see in-between states concerns me because TSM uAPIs would have fully enabled the device to wreak havoc, meanwhile
dev->trustis still showing the device at some lower level of trust. So I think trust modification needs to be synchronous with privileges granted/revoked.If an iommu is present then the device will still be blocked even though it is in RUN, I'm not sure this synchronicity is so important.
Oh, maybe we are just quibbling about where the mechanism lives. The "unblock DMA" step in current preliminary patches is currently behind the "struct
pci_tsm_ops::accept()" op which also handles transitioning the device to RUN / T=1. It is a bus callback.
Quoted 7 lines
Quoted 3 lines
However, if the IOMMU layer is enlightened to block/unblock DMA on trust setting then the TDISP "unblock DMA" step can be factored out of this bus callback and into the IOMMU trust responder.
Yes, I would prefer this because it makes the whole IOMMU mechanism entirely general and not tied to TDISP - which I think is sort of what Greg is pushing on too.
That is what I am going to _require_ here :)
Quoted 31 lines
Quoted 6 lines
I assume this would also expect that encrypted MMIO mappings are also not established while trust is less than "TCB"? That would require some additional enabling to catch attempts to establish an encrypted mapping that the hardware is prepared for, but
dev->trustis not, all without needing to modify the driver to worry about this difference. Drivers would just seeioremap()failure in this case.Hmm.. I don't know if this matters. Once we decide to use the device the MMIO should be mapped in the correct way, whatever that is.
If we decide to eventually allow a lower trust while T=1 then that should be taken to mean the user wants all the features protecting the communication channel but also all the IOMMU features restricting what memory the device can access.
Remember there are two parallel things here, one is T=1 which is designed to protect against hypervisor and physical attacks, the other is the trust level and iommu which would be able to protect against attacks from an attested device itself.
Even if you are in a T=1 environment you may still decide you don't really trust the device firmware that much and would prefer to have it more restricted.
For example, if you have a system with a NVMe drive then all the data on the drive is probably still encrypted and has be CPU-decrypted before it can be used. It would be reasonable to run in T=1 and attest the drive to limit attack surface but also use the IOMMU to limit NVMe access to only the memory used to bounce to the CPU decryption as an additional fortification.
Quoted 4 lines
This is why I am tending to prefer that the kernel's view of trust level and the physical HW capability are somewhat orthogonal things. Even if the HW has high security the user may still prefer that the kernel distrust.
I agree, that's a good way of putting this.
greg k-h
Re: [PATCH v2 03/19] device core: Introduce confidential device acceptance
In reply to Jason GunthorpeJason Gunthorpe wrote:
[..]
Quoted 7 lines
Quoted 6 lines
I assume this would also expect that encrypted MMIO mappings are also not established while trust is less than "TCB"? That would require some additional enabling to catch attempts to establish an encrypted mapping that the hardware is prepared for, but
dev->trustis not, all without needing to modify the driver to worry about this difference. Drivers would just seeioremap()failure in this case.
Quoted 7 lines
Hmm.. I don't know if this matters. Once we decide to use the device the MMIO should be mapped in the correct way, whatever that is.
If we decide to eventually allow a lower trust while T=1 then that should be taken to mean the user wants all the features protecting the communication channel but also all the IOMMU features restricting what memory the device can access.
The question is whether any part of the kernel would ever track that secrets in MMIO writes should not be written to TCB-external devices... but that is probably a "trust=0" situation. "trust=1" means "be careful what you send to this device whether the transport is protected or not".
Quoted 16 lines
Remember there are two parallel things here, one is T=1 which is designed to protect against hypervisor and physical attacks, the other is the trust level and iommu which would be able to protect against attacks from an attested device itself.
Even if you are in a T=1 environment you may still decide you don't really trust the device firmware that much and would prefer to have it more restricted.
For example, if you have a system with a NVMe drive then all the data on the drive is probably still encrypted and has be CPU-decrypted before it can be used. It would be reasonable to run in T=1 and attest the drive to limit attack surface but also use the IOMMU to limit NVMe access to only the memory used to bounce to the CPU decryption as an additional fortification.
Quoted 4 lines
This is why I am tending to prefer that the kernel's view of trust level and the physical HW capability are somewhat orthogonal things. Even if the HW has high security the user may still prefer that the kernel distrust.
Sounds workable to me.
Jason Gunthorpe
maintainerRe: [PATCH v2 03/19] device core: Introduce confidential device acceptance
Quoted 17 lines · Dan Williams
Jason Gunthorpe wrote:
[..]Quoted 7 lines
Quoted 6 lines
I assume this would also expect that encrypted MMIO mappings are also not established while trust is less than "TCB"? That would require some additional enabling to catch attempts to establish an encrypted mapping that the hardware is prepared for, but
dev->trustis not, all without needing to modify the driver to worry about this difference. Drivers would just seeioremap()failure in this case.Quoted 7 lines
Hmm.. I don't know if this matters. Once we decide to use the device the MMIO should be mapped in the correct way, whatever that is.
If we decide to eventually allow a lower trust while T=1 then that should be taken to mean the user wants all the features protecting the communication channel but also all the IOMMU features restricting what memory the device can access.
Quoted 4 lines
The question is whether any part of the kernel would ever track that secrets in MMIO writes should not be written to TCB-external devices... but that is probably a "trust=0" situation. "trust=1" means "be careful what you send to this device whether the transport is protected or not".
Right, the kernel has no idea what is secret or not, it is up to userspace not to give the device secrets if it doesn't fully trust it.
Jason
PATCH v2 4/19 modules: Document the global async_probe parameter
In reply to Dan WilliamsIn preparation for adding another /sys/module/module/parameters entry, document the existing async_probe parameter.
--- Documentation/ABI/stable/sysfs-module | 10 ++++++++++ 1 file changed, 10 insertions(+)
Documentation/ABI/stable/sysfs-module +10 -0
@@ -45,3 +45,13 @@ Date: Jun 2005 Description: If the module source has MODULE_VERSION, this file will contain the version of the source code. + +What: /sys/module/module/parameters/async_probe +Description: + (RW) Emits "1" if drivers from loadable modules attempt async + probing by default. Emits "0" if drivers from loadable modules + attempt synchronous probing by default. This value is overridden + (in priority order) by: the module's built-in "PROBE_FORCE_*" + requests, the "driver_async_probe=..." kernel command line, the + "async_probe" module option, then this default. Write a valid + boolean value to toggle this policy.
Signature · 1 line
2.52.0
PATCH v2 5/19 device core: Autoprobe considered harmful?
In reply to Dan WilliamsThe threat model of PCI Trusted Execution Environment Device Interface Security Protocol (TDISP), is that an adversary may be impersonating the device's identity, redirecting the device's MMIO interface, and/or snooping/manipulating the physical link. Outside of PCI TDISP, PCI ATS (that allows IOMMU bypass) comes to mind as another threat vector that warrants additional device verification beyond whether ACPI enumerates the device as "internal" [1].
The process of verifying a device ranges from the traditional default "accept everything" to gathering signed evidence from a locked device, shipping it to a relying party and acting on that disposition. That policy belongs in userspace. A natural way for userspace to get a control point for verifying a device before use is when the driver for the device comes from a module.
For deployments that are concerned about adversarial devices, introduce a mechanism to disable autoprobe. When a driver originates from a module, consult that driver's autoprobe policy at initial device or driver attach.
Note that with TDISP, unaccepted devices do not have access to private memory (so called "T=0" mode). However, a deployment may still not want to operate more than a handful of boot devices until confirming the system device topology with a verifier.
Yes, this is a security vs convenience tradeoff. Yes, devices with non-modular drivers are out of scope. Yes, there are known regression cases for subsystems where device objects are expected to auto-attach outside of fatal probe failure. For navigating regressions, a per-module "autoprobe" option is included to allow fine grained policy.
--- drivers/base/Kconfig | 24 ++++++++++++++++++++++++ Documentation/ABI/stable/sysfs-module | 10 ++++++++++ drivers/base/base.h | 1 + include/linux/module.h | 14 ++++++++++++++ drivers/base/bus.c | 7 ++++++- drivers/base/dd.c | 26 +++++++++++++++++++++++--- kernel/module/main.c | 13 +++++++++++++ 7 files changed, 91 insertions(+), 4 deletions(-)
drivers/base/Kconfig +24 -0
@@ -253,4 +253,28 @@ config CONFIDENTIAL_DEVICES depends on ARCH_HAS_CC_PLATFORM bool +config MODULES_AUTOPROBE + bool "Automatic probe of drivers from modules" + default y + help + Say Y for the typical and traditional Linux behavior of automatically + attaching devices to drivers when a module is loaded. + + Say N to opt into a threat model where userspace verification of a + device is required before driver attach. This includes Confidential + Computing use cases where the device needs to have its configuration + locked and verified by a relying party. It also includes use cases + like leaving devices with Address Translation (IOMMU protection + bypass) capability disabled until userspace attests the device and + binds a driver. + + This default value can be overridden by the "autoprobe" module option. + Note that some subsystems may not be prepared for autoprobe to be + disabled, take care to test your selected drivers. Built-in drivers are + unaffected by this policy and will autoprobe unless the bus itself has + disabled autoprobe. + + If in doubt, say Y. The N case is only for expert configurations, and + selective "autoprobe=0" in modprobe policy is the common expectation. + endmenu
Documentation/ABI/stable/sysfs-module +10 -0
@@ -55,3 +55,13 @@ Description: requests, the "driver_async_probe=..." kernel command line, the "async_probe" module option, then this default. Write a valid boolean value to toggle this policy. + +What: /sys/module/module/parameters/modules_autoprobe +Description: + (RW) Emits "1" if drivers from loadable modules automatically + attach to their devices. Emits "0" if userspace is responsible + to attach devices to drivers post module load, or device + arrival. This value defaults to CONFIG_MODULES_AUTOPROBE compile + to configuration and is overridden by either a bus's autoprobe + policy or the per-module "autoprobe" option. Write a valid + boolean value to toggle this policy.
drivers/base/base.h +1 -0
@@ -230,6 +230,7 @@ void device_block_probing(void);void device_unblock_probing (void );void deferred_probe_extend_timeout (void );void driver_deferred_probe_trigger (void ); +bool driver_autoprobe (struct device_driver *drv);const char *device_get_devnode (const struct device *dev,umode_t *mode,kuid_t *uid,kgid_t *gid,const char **tmp);
include/linux/module.h +14 -0
@@ -450,6 +450,7 @@ struct module {#endif bool async_probe_requested; +bool autoprobe;/* Exception table */ unsigned int num_exentries;
@@ -761,6 +762,14 @@ static inline bool module_requested_async_probing(struct module *module)return module && module->async_probe_requested; } +static inline bool module_requested_autoprobe (struct module *module) +{ +/* Built-in modules autoprobe by default. */ +if (!module) +return true; +return module->autoprobe; +} +static inline bool is_livepatch_module (struct module *mod) {#ifdef CONFIG_LIVEPATCH
@@ -865,6 +874,11 @@ static inline bool module_requested_async_probing(struct module *module)return false; } +static inline bool module_requested_autoprobe (struct module *module) +{ +/* Built-in modules autoprobe by default. */ +return true; +}static inline void set_module_sig_enforced (void ) {
drivers/base/bus.c +6 -1
@@ -677,6 +677,11 @@ static ssize_t uevent_store(struct device_driver *drv, const char *buf, }static DRIVER_ATTR_WO (uevent); +bool driver_autoprobe (struct device_driver *drv) +{ +return module_requested_autoprobe (drv->owner); +} +/** * bus_add_driver - Add a driver to the bus. * @drv: driver.
@@ -711,7 +716,7 @@ int bus_add_driver(struct device_driver *drv) goto out_unregister;klist_add_tail (&priv->knode_bus, &sp->klist_drivers); -if (sp->drivers_autoprobe) { +if (sp->drivers_autoprobe &&driver_autoprobe (drv)) { error =driver_attach (drv);if (error) goto out_del_list;
drivers/base/dd.c +23 -3
@@ -917,6 +917,12 @@ struct device_attach_data {* driver, we'll encounter one that requests asynchronous probing. */ bool have_async; + +/* +* On initial device arrival driver attach is subject to +* driver_autoprobe() policy. +*/ +bool initial_probe; };static int __device_attach_driver (struct device_driver *drv,void *_data)
@@ -926,6 +932,13 @@ static int __device_attach_driver(struct device_driver *drv, void *_data)bool async_allowed;int ret; +/* +* At initial probe of a newly arrived device, honor the policy to defer +* attachment to explicit userspace bind request. +*/ +if (data->initial_probe && !driver_autoprobe (drv)) +return 0 ; + ret =driver_match_device (drv, dev);if (ret ==0 ) {/* no match */
@@ -998,8 +1011,13 @@ static void __device_attach_async_helper(void *_dev, async_cookie_t cookie)put_device (dev); } -static int __device_attach (struct device *dev,bool allow_async) +#define DEVICE_ATTACH_F_ASYNC BIT(0) +#define DEVICE_ATTACH_F_INITIAL BIT(1) + +static int __device_attach (struct device *dev,unsigned long flags) { +bool allow_async = flags &DEVICE_ATTACH_F_ASYNC ; +bool initial_probe = flags &DEVICE_ATTACH_F_INITIAL ;int ret =0 ;bool async = false;
@@ -1023,6 +1041,7 @@ static int __device_attach(struct device *dev, bool allow_async) .dev = dev, .check_async = allow_async, .want_async = false, + .initial_probe = initial_probe, };if (dev->parent)
@@ -1071,7 +1090,7 @@ static int __device_attach(struct device *dev, bool allow_async)*/ int device_attach (struct device *dev) { -return __device_attach (dev, false); +return __device_attach (dev,0 ); }EXPORT_SYMBOL_GPL (device_attach);
@@ -1083,7 +1102,8 @@ void device_initial_probe(struct device *dev)return ;if (sp->drivers_autoprobe) -__device_attach (dev, true); +__device_attach (dev,DEVICE_ATTACH_F_INITIAL | +DEVICE_ATTACH_F_ASYNC );subsys_put (sp); }
kernel/module/main.c +13 -0
@@ -3001,6 +3001,10 @@ void flush_module_init_free_work(void)static bool async_probe;module_param (async_probe, bool,0644 ); +/* Default value for module->autoprobe */ +bool modules_autoprobe =IS_ENABLED (CONFIG_MODULES_AUTOPROBE ); +module_param (modules_autoprobe, bool,0644 ); +/* * This is where the real work happens. *
@@ -3304,6 +3308,14 @@ static int unknown_module_param_cb(char *param, char *val, const char *modname,return 0 ; } +if (strcmp (param,"autoprobe" ) ==0 ) { +bool autoprobe; + +if (kstrtobool (val, &autoprobe) >=0 ) + mod->autoprobe = autoprobe; +return 0 ; + } +/* Check for magic 'dyndbg' arg */ ret =ddebug_dyndbg_module_param_cb (param, val, modname);if (ret !=0 )
@@ -3473,6 +3485,7 @@ static int load_module(struct load_info *info, const char __user *uargs, goto bug_cleanup; mod->async_probe_requested = async_probe; + mod->autoprobe = modules_autoprobe;/* Module is ready to execute: parsing args may do that. */ after_dashes = parse_args(mod->name, mod->args, mod->kp, mod->num_kp,
Signature · 1 line
2.52.0
Re: [PATCH v2 05/19] device core: Autoprobe considered harmful?
Quoted 30 lines · On Mon, 2 Mar 2026 16:01:53 -0800 Dan Williams <[email protected]>…
The threat model of PCI Trusted Execution Environment Device Interface Security Protocol (TDISP), is that an adversary may be impersonating the device's identity, redirecting the device's MMIO interface, and/or snooping/manipulating the physical link. Outside of PCI TDISP, PCI ATS (that allows IOMMU bypass) comes to mind as another threat vector that warrants additional device verification beyond whether ACPI enumerates the device as "internal" [1].
The process of verifying a device ranges from the traditional default "accept everything" to gathering signed evidence from a locked device, shipping it to a relying party and acting on that disposition. That policy belongs in userspace. A natural way for userspace to get a control point for verifying a device before use is when the driver for the device comes from a module.
For deployments that are concerned about adversarial devices, introduce a mechanism to disable autoprobe. When a driver originates from a module, consult that driver's autoprobe policy at initial device or driver attach.
Note that with TDISP, unaccepted devices do not have access to private memory (so called "T=0" mode). However, a deployment may still not want to operate more than a handful of boot devices until confirming the system device topology with a verifier.
Yes, this is a security vs convenience tradeoff. Yes, devices with non-modular drivers are out of scope. Yes, there are known regression cases for subsystems where device objects are expected to auto-attach outside of fatal probe failure. For navigating regressions, a per-module "autoprobe" option is included to allow fine grained policy.
Quoted 15 lines
Cc Christoph Hellwig <[email protected]>Cc Jason Gunthorpe <[email protected]>Cc Marek Szyprowski <[email protected]>Cc Robin Murphy <[email protected]>Cc Roman Kisel <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Samuel Ortiz <[email protected]>Cc Alexey Kardashevskiy <[email protected]>Cc Xu Yilun <[email protected]>Cc "Aneesh Kumar K.V" <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc "Rafael J. Wysocki" <[email protected]>Cc Danilo Krummrich <[email protected]>Signed-off-by Dan Williams <[email protected]>
Approach seems reasonable to me.
A few trivial things inline.
Jonathan
Quoted 26 lines
--- drivers/base/Kconfig | 24 ++++++++++++++++++++++++ Documentation/ABI/stable/sysfs-module | 10 ++++++++++ drivers/base/base.h | 1 + include/linux/module.h | 14 ++++++++++++++ drivers/base/bus.c | 7 ++++++- drivers/base/dd.c | 26 +++++++++++++++++++++++--- kernel/module/main.c | 13 +++++++++++++ 7 files changed, 91 insertions(+), 4 deletions(-)drivers/base/Kconfig +8
-0@@ -253,4 +253,28 @@ config CONFIDENTIAL_DEVICES depends on ARCH_HAS_CC_PLATFORM bool +config MODULES_AUTOPROBE + bool "Automatic probe of drivers from modules" + default y + help + Say Y for the typical and traditional Linux behavior of automatically + attaching devices to drivers when a module is loaded. + + Say N to opt into a threat model where userspace verification of a
drivers/base/Kconfig config CONFIDENTIAL_DEVICES · line 273
Quoted 10 lines
drivers/base/Kconfig +10
-0+ device is required before driver attach. This includes Confidential + Computing use cases where the device needs to have its configuration + locked and verified by a relying party. It also includes use cases + like leaving devices with Address Translation (IOMMU protection + bypass) capability disabled until userspace attests the device and + binds a driver. + + This default value can be overridden by the "autoprobe" module option. + Note that some subsystems may not be prepared for autoprobe to be + disabled, take care to test your selected drivers. Built-in drivers are
Stray extra space.
Quoted 14 lines
drivers/base/Kconfig +6
-0+ unaffected by this policy and will autoprobe unless the bus itself has + disabled autoprobe. + + If in doubt, say Y. The N case is only for expert configurations, and + selective "autoprobe=0" in modprobe policy is the common expectation. + endmenudrivers/base/dd.c +0
-0
drivers/base/dd.c struct device_attach_data { · line 925
Quoted 9 lines
drivers/base/dd.c +6
-0@@ -917,6 +917,12 @@ struct device_attach_data {* driver, we'll encounter one that requests asynchronous probing. */ bool have_async; + +/* +* On initial device arrival driver attach is subject to +* driver_autoprobe() policy. +*/ +bool initial_probe;
I'm not sure 'initial' naming works. How does that work with drivers that have not autobound anyway? E.g. VFIO.
Seems they'll be fine even if it is their initial probe. Also the deferred cases remain 'initial' for repeated probing.
Why not stick to the auto probing terminology here?
There is clearly history of this naming though in device_initial_probe() So maybe that name is fine...
Quoted 3 lines
drivers/base/dd.c +0
-0};
PATCH v2 6/19 PCI/TSM: Add Device Security (TVM Guest) LOCK operation support
In reply to Dan WilliamsPCIe Trusted Execution Environment Device Interface Security Protocol (TDISP) has two distinct sets of operations. The first, already enabled in driver/pci/tsm.c, enables the VMM to authenticate the physical function (PCIe Component Measurement and Authentication (CMA)), establish a secure message passing session (DMTF SPDM), and establish physical link security (PCIe Integrity and Data Encryption (IDE)). The second set of operations lets a VM manage the security state of assigned devices (TEE Device Interfaces (TDIs)). Implement the LOCK/UNLOCK operations in pci_tsm_ops, to be followed with an ACCEPT operation.
- lock(): Transition the device to the TDISP state. In this mode the device is responsible for validating that it is in a secure configuration and will transition to the TDISP ERROR state if those settings are modified. Device Security Manager (DSM) and the TEE Security Manager (TSM) enforce that the device is not permitted to issue T=1 traffic in this mode.
- unlock(): From the RUN state the only other TDISP states that can be moved to are ERROR or UNLOCKED. Voluntarily move the device to the UNLOCKED state.
--- Documentation/ABI/testing/sysfs-bus-pci | 34 +++- Documentation/ABI/testing/sysfs-class-tsm | 19 ++ Documentation/driver-api/pci/tsm.rst | 44 +++++ include/linux/device.h | 1 + include/linux/pci-tsm.h | 21 ++- drivers/pci/tsm.c | 213 +++++++++++++++++++++- drivers/virt/coco/tsm-core.c | 41 +++++ 7 files changed, 363 insertions(+), 10 deletions(-)
Documentation/ABI/testing/sysfs-bus-pci +32 -2
@@ -647,13 +647,16 @@ Description: Encryption) establishment. Reads from this attribute return the name of the connected TSM or the empty string if not connected. A TSM device signals its readiness to accept PCI - connection via a KOBJ_CHANGE event. + connection via a KOBJ_CHANGE event. This is a "link" TSM + attribute, see Documentation/ABI/testing/sysfs-class-tsm. What: /sys/bus/pci/devices/.../tsm/disconnect Contact: [email protected] Description: (WO) Write the name of the TSM device that was specified - to 'connect' to teardown the connection. + to 'connect' to teardown the connection. This is a + "link" TSM attribute, see + Documentation/ABI/testing/sysfs-class-tsm. What: /sys/bus/pci/devices/.../tsm/dsm Contact: [email protected]
@@ -702,3 +705,30 @@ Description: When present and the tsm/ attribute directory is present, the authenticated attribute is an alias for the device 'connect' state. See the 'tsm/connect' attribute for more details. + This is a "link" TSM attribute, see + Documentation/ABI/testing/sysfs-class-tsm. + +What: /sys/bus/pci/devices/.../tsm/lock +Contact: [email protected] +Description: + (RW) Write the name of a TSM (TEE Security Manager) device + instance to this file to request that the platform transition + the PCIe device configuration to the TDISP LOCKED state. This + puts the device in a state where security sensitive + configuration setting can not be changed without transitioning + the device the PCIe TDISP ERROR state. Reads from this attribute + return the name of the TSM device instance that has arranged for + the device to be locked, or the empty string if not locked. A + TSM class device signals its readiness for lock requests via a + KOBJ_CHANGE event. Writes fail with EBUSY if this device is + bound to a driver. This is a "devsec" TSM attribute, see + Documentation/ABI/testing/sysfs-class-tsm. See + Documentation/driver-api/pci/tsm.rst for more details. + +What: /sys/bus/pci/devices/.../tsm/unlock +Contact: [email protected] +Description: + (WO) Write the name of the TSM device that was specified to + 'lock' to teardown the connection. Writes fail with EBUSY if + this device is bound to a driver. This is a "devsec" TSM + attribute, see Documentation/ABI/testing/sysfs-class-tsm.
Documentation/ABI/testing/sysfs-class-tsm +19 -0
@@ -20,3 +20,22 @@ Description: Documentation/ABI/testing/sysfs-devices-pci-host-bridge for the description of the pciDDDD:BB/streamH.R.E symlink and the pciDDDD:BB/available_secure_streams attribute. + +What: /sys/class/tsm/tsmN/pci_mode +Contact: [email protected] +Description: + (RO) A TSM with PCIe TDISP capability can be in one of two + modes. + + "link": typically for a hypervisor (VMM) to authenticate, + establish a secure session, and setup link + encryption. + + "devsec": typically for a confidential guest (TVM) to + transition assigned devices through the TDISP + state machine UNLOCKED->LOCKED->RUN. + + See the "tsm/" entries in + Documentation/ABI/testing/sysfs-bus-pci for the available PCI + device attributes when a TSM with the given "pci_mode" is + registered.
Documentation/driver-api/pci/tsm.rst +44 -0
50 lines not on this page · raw message
include/linux/device.h +1 -0
@@ -911,6 +911,7 @@ static inline void device_unlock(struct device *dev) } DEFINE_GUARD(device, struct device *, device_lock(_T), device_unlock(_T)) +DEFINE_GUARD_COND(device, _intr, device_lock_interruptible(_T), _RET == 0)static inline void device_lock_assert (struct device *dev) {
include/linux/pci-tsm.h +20 -1
@@ -68,7 +68,8 @@ struct pci_tsm_ops {* @unlock: destroy TSM context and return device to UNLOCKED state * * Context: @lock and @unlock run under pci_tsm_rwsem held for write to -* sync with TSM unregistration and each other +* sync with TSM unregistration and each other. All operations run under +* the device lock for mutual exclusion with driver attach and detach. */ struct_group_tagged (pci_tsm_devsec_ops, devsec_ops, struct pci_tsm *(*lock)(struct tsm_dev *tsm_dev,
@@ -106,6 +107,13 @@ struct pci_tdi {* sub-function (SR-IOV virtual function, or non-function0 * multifunction-device), or a downstream endpoint (PCIe upstream switch-port as * DSM). +* +* For devsec operations it serves to indicate that the function / TDI has been +* locked to a given TSM. +* +* The common expectation is that there is only ever one TSM, but this is not +* enforced. The implementation only enforces that a device can be "connected" +* to a TSM instance or "locked" to a different TSM. */ struct pci_tsm {struct pci_dev *pdev;
@@ -126,6 +134,14 @@ struct pci_tsm_pf0 {struct pci_doe_mb *doe_mb; }; +/** +* struct pci_tsm_devsec - context for tracking private/accepted PCI resources +* @base_tsm: generic core "tsm" context +*/ +struct pci_tsm_devsec { +struct pci_tsm base_tsm; +}; +/* physical function0 and capable of 'connect' */ static inline bool is_pci_tsm_pf0 (struct pci_dev *pdev) {
@@ -206,6 +222,8 @@ int pci_tsm_link_constructor(struct pci_dev *pdev, struct pci_tsm *tsm, struct tsm_dev *tsm_dev);int pci_tsm_pf0_constructor (struct pci_dev *pdev,struct pci_tsm_pf0 *tsm,struct tsm_dev *tsm_dev); +int pci_tsm_devsec_constructor (struct pci_dev *pdev,struct pci_tsm_devsec *tsm, +struct tsm_dev *tsm_dev);void pci_tsm_pf0_destructor (struct pci_tsm_pf0 *tsm);int pci_tsm_doe_transfer (struct pci_dev *pdev,u8 type,const void *req,size_t req_sz,void *resp,size_t resp_sz);
@@ -216,6 +234,7 @@ void pci_tsm_tdi_constructor(struct pci_dev *pdev, struct pci_tdi *tdi,ssize_t pci_tsm_guest_req (struct pci_dev *pdev,enum pci_tsm_req_scope scope,sockptr_t req_in,size_t in_len,sockptr_t req_out,size_t out_len,u64 *tsm_code); +struct pci_tsm_devsec *to_pci_tsm_devsec (struct pci_tsm *tsm);#else static inline int pci_tsm_register (struct tsm_dev *tsm_dev) {
drivers/pci/tsm.c +206 -7
284 lines not on this page · raw message
drivers/virt/coco/tsm-core.c +41 -0
59 lines not on this page · raw message
Signature · 1 line
2.52.0
PATCH v2 7/19 PCI/TSM: Add Device Security (TVM Guest) ACCEPT operation support
In reply to Dan WilliamsThe final operation of the PCIe Trusted Execution Environment (TEE) Device Interface Security Protocol (TDISP) is asking the TEE Security Manager (TEE) to enable private DMA and MMIO.
The story so far in the security lifecycle of the device is that the VMM setup an SPDM session and link encryption with the device's physical function0. The VMM then assigned either that physical function or other virtual function of that device to a VM. The VM asked the TSM to transition the device from TDISP UNLOCKED->LOCKED. With the device LOCKED the VM validated signed fresh device evidence and expected MMIO mappings.
The VM now accepts the device to transition it from LOCKED to RUN and tell the TSM to unblock DMA to VM private memory.
Implement a sysfs trigger to flip the device to private operation and plumb that to a 'struct pci_tsm_ops::accept()' operation.
--- drivers/pci/Kconfig | 2 + Documentation/ABI/testing/sysfs-bus-pci | 13 +++++ include/linux/pci-tsm.h | 7 ++- drivers/pci/tsm.c | 69 ++++++++++++++++++++++++- 4 files changed, 88 insertions(+), 3 deletions(-)
drivers/pci/Kconfig +2 -0
@@ -127,6 +127,8 @@ config PCI_IDE config PCI_TSM bool "PCI TSM: Device security protocol support" + depends on ARCH_HAS_CC_PLATFORM + select CONFIDENTIAL_DEVICES select PCI_IDE select PCI_DOE select TSM
Documentation/ABI/testing/sysfs-bus-pci +13 -0
@@ -732,3 +732,16 @@ Description: 'lock' to teardown the connection. Writes fail with EBUSY if this device is bound to a driver. This is a "devsec" TSM attribute, see Documentation/ABI/testing/sysfs-class-tsm. + +What: /sys/bus/pci/devices/.../tsm/accept +Contact: [email protected] +Description: + (RW) Write "1" (or any boolean "true" string) to this file to + request that TSM transition the device from the TDISP LOCKED + state to the RUN state and arrange the for the secure IOMMU to + accept requests with T=1 in the PCIe packet header (TLP) + targeting private memory. Per TDISP the only exits from the RUN + state are via an explicit unlock request or an event that + transitions the device to the ERROR state. Writes fail with + EBUSY if this device is bound to a driver. This is a "devsec" + TSM attribute, see Documentation/ABI/testing/sysfs-class-tsm.
include/linux/pci-tsm.h +5 -2
@@ -66,15 +66,18 @@ struct pci_tsm_ops {* pci_tsm') for follow-on security state transitions from the * LOCKED state * @unlock: destroy TSM context and return device to UNLOCKED state +* @accept: accept a locked TDI for use, move it to RUN state * * Context: @lock and @unlock run under pci_tsm_rwsem held for write to -* sync with TSM unregistration and each other. All operations run under -* the device lock for mutual exclusion with driver attach and detach. +* sync with TSM unregistration and each other. @accept runs under +* pci_tsm_rwsem held for read. All operations run under the device lock +* for mutual exclusion with driver attach and detach. */ struct_group_tagged (pci_tsm_devsec_ops, devsec_ops, struct pci_tsm *(*lock)(struct tsm_dev *tsm_dev, struct pci_dev *pdev);void (*unlock)(struct pci_tsm *tsm); +int (*accept)(struct pci_dev *pdev); ); };
drivers/pci/tsm.c +68 -1
87 lines not on this page · raw message
Signature · 1 line
2.52.0
Re: [PATCH v2 07/19] PCI/TSM: Add Device Security (TVM Guest) ACCEPT operation support
Quoted 47 lines · Dan Williams
The final operation of the PCIe Trusted Execution Environment (TEE) Device Interface Security Protocol (TDISP) is asking the TEE Security Manager (TEE) to enable private DMA and MMIO.
The story so far in the security lifecycle of the device is that the VMM setup an SPDM session and link encryption with the device's physical function0. The VMM then assigned either that physical function or other virtual function of that device to a VM. The VM asked the TSM to transition the device from TDISP
UNLOCKED->LOCKED. With the device LOCKED the VM validated signed fresh device evidence and expected MMIO mappings.The VM now accepts the device to transition it from LOCKED to RUN and tell the TSM to unblock DMA to VM private memory.
Implement a sysfs trigger to flip the device to private operation and plumb that to a 'struct
pci_tsm_ops::accept()' operation.Co-developed-by Xu Yilun<[email protected]>Signed-off-by Xu Yilun<[email protected]>Co-developed-by Aneesh Kumar K.V (Arm)<[email protected]>Signed-off-by Aneesh Kumar K.V (Arm)<[email protected]>Signed-off-by Dan Williams<[email protected]>--- drivers/pci/Kconfig | 2 + Documentation/ABI/testing/sysfs-bus-pci | 13 +++++ include/linux/pci-tsm.h | 7 ++- drivers/pci/tsm.c | 69 ++++++++++++++++++++++++- 4 files changed, 88 insertions(+), 3 deletions(-)drivers/pci/Kconfig +2
-0@@ -127,6 +127,8 @@ config PCI_IDE config PCI_TSM bool "PCI TSM: Device security protocol support" + depends on ARCH_HAS_CC_PLATFORM + select CONFIDENTIAL_DEVICES select PCI_IDE select PCI_DOE select TSMDocumentation/ABI/testing/sysfs-bus-pci +0
-0
Documentation/ABI/testing/sysfs-bus-pci Description: · line 741
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +7
-0@@ -732,3 +732,16 @@ Description: 'lock' to teardown the connection. Writes fail with EBUSY if this device is bound to a driver. This is a "devsec" TSM attribute, see Documentation/ABI/testing/sysfs-class-tsm. + +What: /sys/bus/pci/devices/.../tsm/accept +Contact: [email protected] +Description: + (RW) Write "1" (or any boolean "true" string) to this file to + request that TSM transition the device from the TDISP LOCKED + state to the RUN state and arrange the for the secure IOMMU to
Nit: remove the typo extra 'the'
"...and arrange for the secure IOMMU to..."
Documentation/ABI/testing/sysfs-bus-pci Description: · line 747
Quoted 6 lines
Documentation/ABI/testing/sysfs-bus-pci +6
-0+ accept requests with T=1 in the PCIe packet header (TLP) + targeting private memory. Per TDISP the only exits from the RUN + state are via an explicit unlock request or an event that + transitions the device to the ERROR state. Writes fail with + EBUSY if this device is bound to a driver. This is a "devsec" + TSM attribute, see Documentation/ABI/testing/sysfs-class-tsm.
Thanks,
baolu
Re: [PATCH v2 07/19] PCI/TSM: Add Device Security (TVM Guest) ACCEPT operation support
In reply to Dan WilliamsQuoted 94 lines · Dan Williams
The final operation of the PCIe Trusted Execution Environment (TEE) Device Interface Security Protocol (TDISP) is asking the TEE Security Manager (TEE) to enable private DMA and MMIO.
The story so far in the security lifecycle of the device is that the VMM setup an SPDM session and link encryption with the device's physical function0. The VMM then assigned either that physical function or other virtual function of that device to a VM. The VM asked the TSM to transition the device from TDISP
UNLOCKED->LOCKED. With the device LOCKED the VM validated signed fresh device evidence and expected MMIO mappings.The VM now accepts the device to transition it from LOCKED to RUN and tell the TSM to unblock DMA to VM private memory.
Implement a sysfs trigger to flip the device to private operation and plumb that to a 'struct
pci_tsm_ops::accept()' operation.Co-developed-by Xu Yilun <[email protected]>Signed-off-by Xu Yilun <[email protected]>Co-developed-by Aneesh Kumar K.V (Arm) <[email protected]>Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>Signed-off-by Dan Williams <[email protected]>--- drivers/pci/Kconfig | 2 + Documentation/ABI/testing/sysfs-bus-pci | 13 +++++ include/linux/pci-tsm.h | 7 ++- drivers/pci/tsm.c | 69 ++++++++++++++++++++++++- 4 files changed, 88 insertions(+), 3 deletions(-)drivers/pci/Kconfig +2
-0@@ -127,6 +127,8 @@ config PCI_IDE config PCI_TSM bool "PCI TSM: Device security protocol support" + depends on ARCH_HAS_CC_PLATFORM + select CONFIDENTIAL_DEVICES select PCI_IDE select PCI_DOE select TSMDocumentation/ABI/testing/sysfs-bus-pci +13
-0@@ -732,3 +732,16 @@ Description: 'lock' to teardown the connection. Writes fail with EBUSY if this device is bound to a driver. This is a "devsec" TSM attribute, see Documentation/ABI/testing/sysfs-class-tsm. + +What: /sys/bus/pci/devices/.../tsm/accept +Contact: [email protected] +Description: + (RW) Write "1" (or any boolean "true" string) to this file to + request that TSM transition the device from the TDISP LOCKED + state to the RUN state and arrange the for the secure IOMMU to + accept requests with T=1 in the PCIe packet header (TLP) + targeting private memory. Per TDISP the only exits from the RUN + state are via an explicit unlock request or an event that + transitions the device to the ERROR state. Writes fail with + EBUSY if this device is bound to a driver. This is a "devsec" + TSM attribute, see Documentation/ABI/testing/sysfs-class-tsm.include/linux/pci-tsm.h +5
-2@@ -66,15 +66,18 @@ struct pci_tsm_ops {* pci_tsm') for follow-on security state transitions from the * LOCKED state * @unlock: destroy TSM context and return device to UNLOCKED state +* @accept: accept a locked TDI for use, move it to RUN state * * Context: @lock and @unlock run under pci_tsm_rwsem held for write to -* sync with TSM unregistration and each other. All operations run under -* the device lock for mutual exclusion with driver attach and detach. +* sync with TSM unregistration and each other. @accept runs under +* pci_tsm_rwsem held for read. All operations run under the device lock +* for mutual exclusion with driver attach and detach. */ struct_group_tagged (pci_tsm_devsec_ops, devsec_ops, struct pci_tsm *(*lock)(struct tsm_dev *tsm_dev, struct pci_dev *pdev);void (*unlock)(struct pci_tsm *tsm); +int (*accept)(struct pci_dev *pdev); ); };drivers/pci/tsm.c +0
-0@@ -557,6 +557,71 @@ static ssize_t dsm_show(struct device *dev, struct device_attribute *attr, }
drivers/pci/tsm.c static ssize_t dsm_show(struct device *dev, struct device_attribute *attr, · line 567
Quoted 10 lines
drivers/pci/tsm.c +8
-0static DEVICE_ATTR_RO (dsm); +/** +* pci_tsm_accept() - accept a device for private MMIO+DMA operation +* @pdev: PCI device to accept +* +* "Accept" transitions a device to the run state, it is only suitable to make +* that transition from a known DMA-idle (no active mappings) state. The "driver +* detached" state is a coarse way to assert that requirement. +*/
Hi Dan,
Repeated accept on a device that is already in RUN state is not rejected by the PCI TSM core, and multiple encrypted MMIO resources for the same BAR range can be created. Furthermore, a later request to move the device to UNLOCKED state only removes the most recently tracked encrypted range.
Reproduce steps:
1. echo tsmX > /sys/bus/pci/devices/<bdf>/tsm/lock
2. echo 1 > /sys/bus/pci/devices/<bdf>/tsm/accept
3. echo 1 > /sys/bus/pci/devices/<bdf>/tsm/accept
4. cat /proc/iomem | grep "PCI MMIO Encrypted"
5. echo tsmX > /sys/bus/pci/devices/<bdf>/tsm/unlock
6. cat /proc/iomem | grep "PCI MMIO Encrypted"
Observed results after step4 (duplicate BAR range):
380002000000-3800021fffff : PCI MMIO Encrypted
380002000000-3800021fffff : PCI MMIO Encrypted
Observed results after step 6 (leaked resource):
380002000000-3800021fffff : PCI MMIO Encrypted
Regards,
Yi Lai
Quoted 81 lines
drivers/pci/tsm.c +60
-1+static int pci_tsm_accept (struct pci_dev *pdev) +{ +int rc; + +ACQUIRE (rwsem_read_intr, lock)(&pci_tsm_rwsem); +if ((rc =ACQUIRE_ERR (rwsem_read_intr, &lock))) +return rc; + +if (!pdev->tsm) +return -EINVAL ; + +ACQUIRE (device_intr, dev_lock)(&pdev->dev); +if ((rc =ACQUIRE_ERR (device_intr, &dev_lock))) +return rc; + +if (pdev->dev.driver) +return -EBUSY ; + + rc =to_pci_tsm_ops (pdev->tsm)->accept (pdev); +if (rc) +return rc; + +return device_cc_accept (&pdev->dev); +} + +static ssize_t accept_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct pci_dev *pdev =to_pci_dev (dev); +bool accept; +int rc; + + rc =kstrtobool (buf, &accept); +if (rc) +return rc; + +/* +* TDISP can only go from RUN to UNLOCKED/ERROR, so there is no +* 'unaccept' verb. +*/ +if (!accept) +return -EINVAL ; + + rc =pci_tsm_accept (pdev); +if (rc) +return rc; + +return len; +} + +static ssize_t accept_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +return sysfs_emit (buf,"%d\n" ,device_cc_accepted (dev)); +} +static DEVICE_ATTR_RW (accept); +/** * pci_tsm_unlock() - Transition TDI from LOCKED/RUN to UNLOCKED * @pdev: TDI device to unlock @@ -740,7 +805,8 @@ static umode_t pci_tsm_attr_visible(struct kobject *kobj, }if (pci_tsm_devsec_group_visible (kobj)) { -if (attr == &dev_attr_lock.attr || +if (attr == &dev_attr_accept.attr || + attr == &dev_attr_lock.attr || attr == &dev_attr_unlock.attr)return attr->mode; }@@ -760,6 +826,7 @@ static struct attribute *pci_tsm_attrs[] = { &dev_attr_disconnect.attr, &dev_attr_bound.attr, &dev_attr_dsm.attr, + &dev_attr_accept.attr, &dev_attr_lock.attr, &dev_attr_unlock.attr,NULL Signature · 1 line
2.52.0
Re: [PATCH v2 07/19] PCI/TSM: Add Device Security (TVM Guest) ACCEPT operation support
In reply to Dan WilliamsQuoted 119 lines · Dan Williams
The final operation of the PCIe Trusted Execution Environment (TEE) Device Interface Security Protocol (TDISP) is asking the TEE Security Manager (TEE) to enable private DMA and MMIO.
The story so far in the security lifecycle of the device is that the VMM setup an SPDM session and link encryption with the device's physical function0. The VMM then assigned either that physical function or other virtual function of that device to a VM. The VM asked the TSM to transition the device from TDISP
UNLOCKED->LOCKED. With the device LOCKED the VM validated signed fresh device evidence and expected MMIO mappings.The VM now accepts the device to transition it from LOCKED to RUN and tell the TSM to unblock DMA to VM private memory.
Implement a sysfs trigger to flip the device to private operation and plumb that to a 'struct
pci_tsm_ops::accept()' operation.Co-developed-by Xu Yilun <[email protected]>Signed-off-by Xu Yilun <[email protected]>Co-developed-by Aneesh Kumar K.V (Arm) <[email protected]>Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>Signed-off-by Dan Williams <[email protected]>--- drivers/pci/Kconfig | 2 + Documentation/ABI/testing/sysfs-bus-pci | 13 +++++ include/linux/pci-tsm.h | 7 ++- drivers/pci/tsm.c | 69 ++++++++++++++++++++++++- 4 files changed, 88 insertions(+), 3 deletions(-)drivers/pci/Kconfig +2
-0@@ -127,6 +127,8 @@ config PCI_IDE config PCI_TSM bool "PCI TSM: Device security protocol support" + depends on ARCH_HAS_CC_PLATFORM + select CONFIDENTIAL_DEVICES select PCI_IDE select PCI_DOE select TSMDocumentation/ABI/testing/sysfs-bus-pci +13
-0@@ -732,3 +732,16 @@ Description: 'lock' to teardown the connection. Writes fail with EBUSY if this device is bound to a driver. This is a "devsec" TSM attribute, see Documentation/ABI/testing/sysfs-class-tsm. + +What: /sys/bus/pci/devices/.../tsm/accept +Contact: [email protected] +Description: + (RW) Write "1" (or any boolean "true" string) to this file to + request that TSM transition the device from the TDISP LOCKED + state to the RUN state and arrange the for the secure IOMMU to + accept requests with T=1 in the PCIe packet header (TLP) + targeting private memory. Per TDISP the only exits from the RUN + state are via an explicit unlock request or an event that + transitions the device to the ERROR state. Writes fail with + EBUSY if this device is bound to a driver. This is a "devsec" + TSM attribute, see Documentation/ABI/testing/sysfs-class-tsm.include/linux/pci-tsm.h +5
-2@@ -66,15 +66,18 @@ struct pci_tsm_ops {* pci_tsm') for follow-on security state transitions from the * LOCKED state * @unlock: destroy TSM context and return device to UNLOCKED state +* @accept: accept a locked TDI for use, move it to RUN state * * Context: @lock and @unlock run under pci_tsm_rwsem held for write to -* sync with TSM unregistration and each other. All operations run under -* the device lock for mutual exclusion with driver attach and detach. +* sync with TSM unregistration and each other. @accept runs under +* pci_tsm_rwsem held for read. All operations run under the device lock +* for mutual exclusion with driver attach and detach. */ struct_group_tagged (pci_tsm_devsec_ops, devsec_ops, struct pci_tsm *(*lock)(struct tsm_dev *tsm_dev, struct pci_dev *pdev);void (*unlock)(struct pci_tsm *tsm); +int (*accept)(struct pci_dev *pdev); ); };drivers/pci/tsm.c +23
-0@@ -557,6 +557,71 @@ static ssize_t dsm_show(struct device *dev, struct device_attribute *attr, }static DEVICE_ATTR_RO (dsm); +/** +* pci_tsm_accept() - accept a device for private MMIO+DMA operation +* @pdev: PCI device to accept +* +* "Accept" transitions a device to the run state, it is only suitable to make +* that transition from a known DMA-idle (no active mappings) state. The "driver +* detached" state is a coarse way to assert that requirement. +*/ +static int pci_tsm_accept (struct pci_dev *pdev) +{ +int rc; + +ACQUIRE (rwsem_read_intr, lock)(&pci_tsm_rwsem); +if ((rc =ACQUIRE_ERR (rwsem_read_intr, &lock))) +return rc; + +if (!pdev->tsm) +return -EINVAL ; + +ACQUIRE (device_intr, dev_lock)(&pdev->dev); +if ((rc =ACQUIRE_ERR (device_intr, &dev_lock))) +return rc; +
drivers/pci/tsm.c static ssize_t dsm_show(struct device *dev, struct device_attribute *attr, · line 592
Quoted 10 lines
drivers/pci/tsm.c +10
-0+if (pdev->dev.driver) +return -EBUSY ; + + rc =to_pci_tsm_ops (pdev->tsm)->accept (pdev); +if (rc) +return rc; + +return device_cc_accept (&pdev->dev); +} +
# Re-send to Dan's kernel.org address. Sorry if you receive the same
# email twice.
Repeated accept on a device that is already in RUN state is not rejected by the PCI TSM core, and multiple encrypted MMIO resources for the same BAR range can be created. Furthermore, a later request to move the device to UNLOCKED state only removes the most recently tracked encrypted range.
Reproduce steps:
1. echo tsmX > /sys/bus/pci/devices/<bdf>/tsm/lock
2. echo 1 > /sys/bus/pci/devices/<bdf>/tsm/accept
3. echo 1 > /sys/bus/pci/devices/<bdf>/tsm/accept
4. cat /proc/iomem | grep "PCI MMIO Encrypted"
5. echo tsmX > /sys/bus/pci/devices/<bdf>/tsm/unlock
6. cat /proc/iomem | grep "PCI MMIO Encrypted"
Observed results after step4 (duplicate BAR range):
380002000000-3800021fffff : PCI MMIO Encrypted
380002000000-3800021fffff : PCI MMIO Encrypted
Observed results after step 6 (leaked resource):
380002000000-3800021fffff : PCI MMIO Encrypted
Regards,
Yi Lai
Quoted 56 lines
drivers/pci/tsm.c +35
-1+static ssize_t accept_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct pci_dev *pdev =to_pci_dev (dev); +bool accept; +int rc; + + rc =kstrtobool (buf, &accept); +if (rc) +return rc; + +/* +* TDISP can only go from RUN to UNLOCKED/ERROR, so there is no +* 'unaccept' verb. +*/ +if (!accept) +return -EINVAL ; + + rc =pci_tsm_accept (pdev); +if (rc) +return rc; + +return len; +} + +static ssize_t accept_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +return sysfs_emit (buf,"%d\n" ,device_cc_accepted (dev)); +} +static DEVICE_ATTR_RW (accept); +/** * pci_tsm_unlock() - Transition TDI from LOCKED/RUN to UNLOCKED * @pdev: TDI device to unlock @@ -740,7 +805,8 @@ static umode_t pci_tsm_attr_visible(struct kobject *kobj, }if (pci_tsm_devsec_group_visible (kobj)) { -if (attr == &dev_attr_lock.attr || +if (attr == &dev_attr_accept.attr || + attr == &dev_attr_lock.attr || attr == &dev_attr_unlock.attr)return attr->mode; }@@ -760,6 +826,7 @@ static struct attribute *pci_tsm_attrs[] = { &dev_attr_disconnect.attr, &dev_attr_bound.attr, &dev_attr_dsm.attr, + &dev_attr_accept.attr, &dev_attr_lock.attr, &dev_attr_unlock.attr,NULL Signature · 1 line
2.52.0
PATCH v2 8/19 PCI/TSM: Add "evidence" support
In reply to Dan WilliamsOnce one accepts the threat model that devices may be adversarial the process of establishing trust in the device identity, the integrity + confidentiality of its link, and the integrity + confidentiality of its MMIO interface requires multiple evidence objects from the device. The device's certificate chain, measurements and interface report need to be retrieved by the host, validated by the TSM and transmitted to the guest all while mitigating TOCTOU races.
All TSM implementations share the same fundamental objects, but vary in how the TSM conveys its trust in the objects. Some TSM implementations expect the full documents to be conveyed over untrustworthy channels while the TSM securely conveys a digest. Others transmit full objects with signed SPDM transcripts of requester provided nonces. Some offer a single transcript to convey the version, capabilities, and algorithms (VCA) data and measurements in one blob while others split VCA as a separate signed blob.
Introduce a netlink interface to dump all these objects in a common way across TSM implementations and across host and guest environments. Userspace is responsible for handling the variance of "TSM provides combo measurements + VCA + nonce + signature, vs TSM provides a digest over a secure channel of the same".
The implementation adheres to the guideline from:Documentation/userspace-api/netlink/genetlink-legacy.rst
New Netlink families should never respond to a DO operation with multiple replies, with ``NLM_F_MULTI`` set. Use a filtered dump instead.
Per SPDM, transcripts may grow to be 16MB in size. Large PCI/TSM netlink blobs are handled via a sequence of dump messages that userspace must concatenate.
---
drivers/pci/Makefile | 2 +-
drivers/pci/tsm/Makefile | 9 +
Documentation/netlink/specs/pci-tsm.yaml | 151 +++++++++++++
drivers/pci/tsm/netlink.h | 23 ++
include/linux/pci-tsm.h | 63 ++++++
include/uapi/linux/pci-tsm-netlink.h | 101 +++++++++
drivers/pci/{tsm.c => tsm/core.c} | 17 +-
drivers/pci/tsm/evidence.c | 274 +++++++++++++++++++++++
drivers/pci/tsm/netlink.c | 43 ++++
MAINTAINERS | 4 +-
10 files changed, 682 insertions(+), 5 deletions(-)
create mode 100644 drivers/pci/tsm/Makefile
create mode 100644 Documentation/netlink/specs/pci-tsm.yaml
create mode 100644 drivers/pci/tsm/netlink.h
create mode 100644 include/uapi/linux/pci-tsm-netlink.h
rename drivers/pci/{tsm.c => tsm/core.c} (98%)
create mode 100644 drivers/pci/tsm/evidence.c
create mode 100644 drivers/pci/tsm/netlink.cdrivers/pci/Makefile +1 -1
@@ -35,7 +35,7 @@ obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o obj-$(CONFIG_VGA_ARB) += vgaarb.o obj-$(CONFIG_PCI_DOE) += doe.o obj-$(CONFIG_PCI_IDE) += ide.o -obj-$(CONFIG_PCI_TSM) += tsm.o +obj-$(CONFIG_PCI_TSM) += tsm/ obj-$(CONFIG_PCI_DYNAMIC_OF_NODES) += of_property.o obj-$(CONFIG_PCI_NPEM) += npem.o obj-$(CONFIG_PCIE_TPH) += tph.o
drivers/pci/tsm/Makefile new file +9 -0
@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for the PCI/TSM infrastructure + +obj-$(CONFIG_PCI_TSM) += tsm.o + +tsm-y := core.o +tsm-$(CONFIG_NET) += netlink.o +tsm-$(CONFIG_NET) += evidence.o
Documentation/netlink/specs/pci-tsm.yaml new file +151 -0
151 lines not on this page · raw message
drivers/pci/tsm/netlink.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */ +/* Do not edit directly, auto-generated from: */ +/* Documentation/netlink/specs/pci-tsm.yaml */ +/* YNL-GEN kernel header */ +/* To regenerate run: tools/net/ynl/ynl-regen.sh */ + +#ifndef _LINUX_PCI_TSM_GEN_H +#define _LINUX_PCI_TSM_GEN_H + +#include <net/netlink.h> +#include <net/genetlink.h> + +#include <uapi/linux/pci-tsm-netlink.h> + +int pci_tsm_nl_evidence_read_pre (struct netlink_callback *cb); +int pci_tsm_nl_evidence_read_post (struct netlink_callback *cb); + +int pci_tsm_nl_evidence_read_dumpit (struct sk_buff *skb, +struct netlink_callback *cb); + +extern struct genl_family pci_tsm_nl_family; + +#endif /* _LINUX_PCI_TSM_GEN_H */
include/linux/pci-tsm.h +63 -0
104 lines not on this page · raw message
include/uapi/linux/pci-tsm-netlink.h new file +101 -0
101 lines not on this page · raw message
drivers/pci/tsm/core.c renamed from drivers/pci/tsm.c +14 -3
@@ -3,7 +3,7 @@* Interface with platform TEE Security Manager (TSM) objects as defined by * PCIe r7.0 section 11 TEE Device Interface Security Protocol (TDISP) * -* Copyright(c) 2024-2025 Intel Corporation. All rights reserved. +* Copyright (C) 2024-2026 Intel Corporation */ #define dev_fmt (fmt) "PCI/TSM: " fmt
@@ -16,13 +16,13 @@#include <linux/sysfs.h> #include <linux/tsm.h> #include <linux/xarray.h> -#include "pci.h" +#include "../pci.h" /* * Provide a read/write lock against the init / exit of pdev tsm * capabilities and arrival/departure of a TSM instance */ -static DECLARE_RWSEM (pci_tsm_rwsem); +DECLARE_RWSEM (pci_tsm_rwsem);/* * Count of TSMs registered that support physical link operations vs device
@@ -302,6 +302,7 @@ static ssize_t connect_store(struct device *dev, struct device_attribute *attr, rc =pci_tsm_connect (pdev, tsm_dev);if (rc)return rc; +return len; }static DEVICE_ATTR_RW (connect);
@@ -933,6 +934,16 @@ void pci_tsm_tdi_constructor(struct pci_dev *pdev, struct pci_tdi *tdi, }EXPORT_SYMBOL_GPL (pci_tsm_tdi_constructor); +void pci_tsm_init_evidence (struct pci_tsm_evidence *evidence,int slot, +enum hash_algo digest_algo) +{ + evidence->slot = slot; + evidence->generation =1 ; + evidence->digest_algo = digest_algo; +init_rwsem (&evidence->lock); +} +EXPORT_SYMBOL_GPL (pci_tsm_init_evidence); +/** * pci_tsm_link_constructor() - base 'struct pci_tsm' initialization for link TSMs * @pdev: The PCI device
drivers/pci/tsm/evidence.c new file +274 -0
274 lines not on this page · raw message
drivers/pci/tsm/netlink.c new file +43 -0
43 lines not on this page · raw message
MAINTAINERS +3 -1
@@ -26534,9 +26534,11 @@ S: Maintained F: Documentation/ABI/testing/configfs-tsm-report F: Documentation/driver-api/coco/ F: Documentation/driver-api/pci/tsm.rst -F: drivers/pci/tsm.c +F: Documentation/netlink/specs/pci-tsm.yaml +F: drivers/pci/tsm/ F: drivers/virt/coco/guest/ F: include/linux/*tsm*.h +F: include/uapi/linux/pci-tsm-netlink.h F: samples/tsm-mr/ TRUSTED SERVICES TEE DRIVER
Signature · 1 line
2.52.0
kernel test robot
build botRe: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
kernel test robot noticed the following build warnings:
Hi Dan,
kernel test robot noticed the following build warnings:
[auto build test WARNING on c2012263047689e495e81c96d7d5b0586299578d]
url: https://github.com/intel-lab-lkp/linux/commits/Dan-Williams/PCI-TSM-Report-active-IDE-streams-per-host-bridge/20260303-080409 base: c2012263047689e495e81c96d7d5b0586299578d patch link: https://lore.kernel.org/r/20260303000207.1836586-9-dan.j.williams%40intel.com patch subject: [PATCH v2 08/19] PCI/TSM: Add "evidence" support compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) docutils: docutils (Docutils 0.21.2, Python 3.13.5, on linux) reproduce: (https://download.01.org/0day-ci/archive/20260303/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
Warning: tools/docs/documentation-file-ref-check references a file that doesn't exist: m,^Documentation/scheduler/sched-pelt
Warning: tools/docs/documentation-file-ref-check references a file that doesn't exist: m,(Documentation/translations/[ Using alabaster theme
WARNING: ./include/crypto/skcipher.h:166 struct member 'SKCIPHER_ALG_COMMON' not described in 'skcipher_alg' Documentation/driver-api/pci/tsm:147: ./include/linux/pci-tsm.h:112: ERROR: Unexpected indentation. [docutils] >> Documentation/driver-api/pci/tsm:147: ./include/linux/pci-tsm.h:113: WARNING: Block quote ends without a blank line; unexpected unindent. [docutils] >> Documentation/driver-api/pci/tsm:147: ./include/linux/pci-tsm.h:116: WARNING: Definition list ends without a blank line; unexpected unindent. [docutils] ERROR: Cannot find file ./drivers/pci/tsm.c ERROR: Cannot find file ./drivers/pci/tsm.c WARNING: No kernel-doc for file ./drivers/pci/tsm.c WARNING: ./include/linux/virtio.h:183 struct member 'map' not described in 'virtio_device' WARNING: ./include/linux/virtio.h:183 struct member 'VIRTIO_DECLARE_FEATURES(features' not described in 'virtio_device'
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Aneesh Kumar K.V
maintainerRe: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
In reply to Dan WilliamsQuoted 145 lines · Dan Williams
Once one accepts the threat model that devices may be adversarial the process of establishing trust in the device identity, the integrity + confidentiality of its link, and the integrity + confidentiality of its MMIO interface requires multiple evidence objects from the device. The device's certificate chain, measurements and interface report need to be retrieved by the host, validated by the TSM and transmitted to the guest all while mitigating TOCTOU races.
All TSM implementations share the same fundamental objects, but vary in how the TSM conveys its trust in the objects. Some TSM implementations expect the full documents to be conveyed over untrustworthy channels while the TSM securely conveys a digest. Others transmit full objects with signed SPDM transcripts of requester provided nonces. Some offer a single transcript to convey the version, capabilities, and algorithms (VCA) data and measurements in one blob while others split VCA as a separate signed blob.
Introduce a netlink interface to dump all these objects in a common way across TSM implementations and across host and guest environments. Userspace is responsible for handling the variance of "TSM provides combo measurements + VCA + nonce + signature, vs TSM provides a digest over a secure channel of the same".
The implementation adheres to the guideline from:
Documentation/userspace-api/netlink/genetlink-legacy.rstNew Netlink families should never respond to a DO operation with multiple replies, with ``NLM_F_MULTI`` set. Use a filtered dump instead.
Per SPDM, transcripts may grow to be 16MB in size. Large PCI/TSM netlink blobs are handled via a sequence of dump messages that userspace must concatenate.
Cc Donald Hunter <[email protected]>Cc Jakub Kicinski <[email protected]>Cc Bjorn Helgaas <[email protected]>Cc Xu Yilun <[email protected]>Cc "Aneesh Kumar K.V (Arm)" <[email protected]>Cc Alexey Kardashevskiy <[email protected]>Signed-off-by Dan Williams <[email protected]>--- drivers/pci/Makefile | 2 +- drivers/pci/tsm/Makefile | 9 + Documentation/netlink/specs/pci-tsm.yaml | 151 +++++++++++++ drivers/pci/tsm/netlink.h | 23 ++ include/linux/pci-tsm.h | 63 ++++++ include/uapi/linux/pci-tsm-netlink.h | 101 +++++++++ drivers/pci/{tsm.c => tsm/core.c} | 17 +- drivers/pci/tsm/evidence.c | 274 +++++++++++++++++++++++ drivers/pci/tsm/netlink.c | 43 ++++ MAINTAINERS | 4 +- 10 files changed, 682 insertions(+), 5 deletions(-) create mode 100644 drivers/pci/tsm/Makefile create mode 100644 Documentation/netlink/specs/pci-tsm.yaml create mode 100644 drivers/pci/tsm/netlink.h create mode 100644 include/uapi/linux/pci-tsm-netlink.h rename drivers/pci/{tsm.c => tsm/core.c} (98%) create mode 100644 drivers/pci/tsm/evidence.c create mode 100644 drivers/pci/tsm/netlink.cdrivers/pci/Makefile +1
-1@@ -35,7 +35,7 @@ obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o obj-$(CONFIG_VGA_ARB) += vgaarb.o obj-$(CONFIG_PCI_DOE) += doe.o obj-$(CONFIG_PCI_IDE) += ide.o -obj-$(CONFIG_PCI_TSM) += tsm.o +obj-$(CONFIG_PCI_TSM) += tsm/ obj-$(CONFIG_PCI_DYNAMIC_OF_NODES) += of_property.o obj-$(CONFIG_PCI_NPEM) += npem.o obj-$(CONFIG_PCIE_TPH) += tph.odrivers/pci/tsm/Makefile new file +9
-0@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for the PCI/TSM infrastructure + +obj-$(CONFIG_PCI_TSM) += tsm.o + +tsm-y := core.o +tsm-$(CONFIG_NET) += netlink.o +tsm-$(CONFIG_NET) += evidence.oDocumentation/netlink/specs/pci-tsm.yaml new file +50
-0@@ -0,0 +1,151 @@ +# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) +# +--- +name: pci-tsm +protocol: genetlink +uapi-header: linux/pci-tsm-netlink.h +doc: PCI TSM Evidence retrieval over generic netlink + +definitions: + - + type: const + name: max-object-size + value: 0x01000000 + - + type: const + name: max-nonce-size + value: 256 + - + type: const + name: max-obj-type + value: 4 + - + name: evidence-type + type: enum + doc: PCI device security evidence objects + entries: + - + name: cert0 + doc: SPDM certificate chain from device slot0 + - + name: cert1 + doc: SPDM certificate chain from device slot1 + - + name: cert2 + doc: SPDM certificate chain from device slot2 + - + name: cert3 + doc: SPDM certificate chain from device slot3 + - + name: cert4 + doc: SPDM certificate chain from device slot4 + - + name: cert5 + doc: SPDM certificate chain from device slot5 + - + name: cert6 + doc: SPDM certificate chain from device slot6 + - + name: cert7 + doc: SPDM certificate chain from device slot7
Documentation/netlink/specs/pci-tsm.yaml line 60
Quoted 10 lines
Documentation/netlink/specs/pci-tsm.yaml new file +10
-0+ - + name: vca + doc: SPDM transcript of version, capabilities, and algorithms negotiation + - + name: measurements + doc: SPDM GET_MEASUREMENTS response + - + name: report + doc: TDISP GET_DEVICE_INTERFACE_REPORT response +
In the case of CCA, the slot number is determined early, when we create the pdev object that maps to PF0. This is done as part of the connect callback. Currently, the slot number is hardcoded to 0. I believe we need to extend connect to include slot information.
Even with that change, we would only have one certificate type. These would correspond to whichever slot number was selected during connect.
-aneesh
Aneesh Kumar K.V
maintainerRe: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
In reply to Dan WilliamsQuoted 25 lines · Dan Williams
Once one accepts the threat model that devices may be adversarial the process of establishing trust in the device identity, the integrity + confidentiality of its link, and the integrity + confidentiality of its MMIO interface requires multiple evidence objects from the device. The device's certificate chain, measurements and interface report need to be retrieved by the host, validated by the TSM and transmitted to the guest all while mitigating TOCTOU races.
All TSM implementations share the same fundamental objects, but vary in how the TSM conveys its trust in the objects. Some TSM implementations expect the full documents to be conveyed over untrustworthy channels while the TSM securely conveys a digest. Others transmit full objects with signed SPDM transcripts of requester provided nonces. Some offer a single transcript to convey the version, capabilities, and algorithms (VCA) data and measurements in one blob while others split VCA as a separate signed blob.
Introduce a netlink interface to dump all these objects in a common way across TSM implementations and across host and guest environments. Userspace is responsible for handling the variance of "TSM provides combo measurements + VCA + nonce + signature, vs TSM provides a digest over a secure channel of the same".
The implementation adheres to the guideline from:
Documentation/userspace-api/netlink/genetlink-legacy.rst
Quoted 8 lines
New Netlink families should never respond to a DO operation with multiple replies, with ``NLM_F_MULTI`` set. Use a filtered dump instead.
Per SPDM, transcripts may grow to be 16MB in size. Large PCI/TSM netlink blobs are handled via a sequence of dump messages that userspace must concatenate.
Should we also expose evidence->generation to userspace so it can be used during accept()? This would allow us to ensure that the device is accepted using the same evidence generation observed by userspace.
-aneesh
Re: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
In reply to Dan WilliamsReview
Quoted 7 lines
(quoted diff) +7
-0+void pci_tsm_init_evidence(struct pci_tsm_evidence *evidence, int slot, + enum hash_algo digest_algo) +{ + evidence->slot = slot; + evidence->generation = 1; + evidence->digest_algo = digest_algo; + init_rwsem(&evidence->lock);
IIUC, this function is for link tsm driver, is it? But in the following patch, devsec tsm would consume pci_tsm_mmio_alloc() which uses evidence->lock. So my solution is to initialize the lock on tsm construction.
--------8<--------
drivers/pci/tsm/core.c +2 -1
@@ -1175,7 +1175,6 @@ void pci_tsm_init_evidence(struct pci_tsm_evidence *evidence, int slot, evidence->slot = slot; evidence->generation =1 ; evidence->digest_algo = digest_algo; -init_rwsem (&evidence->lock); }EXPORT_SYMBOL_GPL (pci_tsm_init_evidence);
@@ -1198,6 +1197,7 @@ int pci_tsm_link_constructor(struct pci_dev *pdev, struct pci_tsm *tsm, } tsm->pdev = pdev; tsm->tsm_dev = tsm_dev; +init_rwsem (&tsm->evidence.lock);return 0 ; }
@@ -1221,6 +1221,7 @@ int pci_tsm_devsec_constructor(struct pci_dev *pdev, struct pci_tsm_devsec *tsm, pci_tsm->tdi =NULL ; pci_tsm->pdev = pdev; pci_tsm->tsm_dev = tsm_dev; +init_rwsem (&pci_tsm->evidence.lock);return 0 ; }
Re: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
Quoted 9 lines · Xu Yilun
Review
Quoted 7 lines
(quoted diff) +7
-0+void pci_tsm_init_evidence(struct pci_tsm_evidence *evidence, int slot, + enum hash_algo digest_algo) +{ + evidence->slot = slot; + evidence->generation = 1; + evidence->digest_algo = digest_algo; + init_rwsem(&evidence->lock);IIUC, this function is for link tsm driver, is it?
It is meant to be generic for both, and an "optional" support library for the low-level TSM drivers.
- Host PCI/TSM evidence interface collects the blobs
- Guest PCI/TSM evidence interface retrieves the digests via private TSM GHCI
- Some infrastructure (either arch specific GHCI or new common GHCI) pushes the blobs from Host to Guest so that guest PCI/TSM evidence
gathering can also get the blobs.
Quoted 3 lines
But in the following patch, devsec tsm would consume
pci_tsm_mmio_alloc()which usesevidence->lock. So my solution is to initialize the lock on tsm construction.
So I did flub the "->evidence == 0" check, and yes initializing the lock by default looks like the right answer to that problem.
Jakub Kicinski
maintainerRe: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
In reply to Dan WilliamsQuoted 6 lines · On Mon, 2 Mar 2026 16:01:56 -0800 Dan Williams wrote:
The implementation adheres to the guideline from:
Documentation/userspace-api/netlink/genetlink-legacy.rstNew Netlink families should never respond to a DO operation with multiple replies, with ``NLM_F_MULTI`` set. Use a filtered dump instead.
My understanding of F_MULTI is that deserializer is supposed to continue deserializing into current object. IOW if we have:
struct does_this {
int really;
int have_to;
int be_netlink;
};You can send "really" and "be_netlink" in one message and "have_to" in the next, and receiver should reconstruct them into a single struct.
If F_MULTI is not set - receiver assumes that the next message is a new struct. And the whole dump returns a list of structs.
So IOW I think what you're doing is a bit too.. inventive. Do you have plans to add more commands?
The read-only stuff feels like it could be a sysfs API? The main strength of Netlink is "do" commands with multiple optional attrs.
Re: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
Quoted 8 lines · Jakub Kicinski
Quoted 6 lines · On Mon, 2 Mar 2026 16:01:56 -0800 Dan Williams wrote:
The implementation adheres to the guideline from:
Documentation/userspace-api/netlink/genetlink-legacy.rstNew Netlink families should never respond to a DO operation with multiple replies, with ``NLM_F_MULTI`` set. Use a filtered dump instead.
Quoted 8 lines
My understanding of
F_MULTIis that deserializer is supposed to continue deserializing into current object. IOW if we have:struct does_this { int really; int have_to; int be_netlink; };
Heh, sensing a subtle message here...
Quoted 7 lines
You can send "really" and "
be_netlink" in one message and "have_to" in the next, and receiver should reconstruct them into a single struct.If
F_MULTIis not set - receiver assumes that the next message is a new struct. And the whole dump returns a list of structs.So IOW I think what you're doing is a bit too.. inventive.
Fair, but see below, satisfying the requirements here are stuck in the liminal space between sysfs and netlink...
Quoted 1 line
Do you have plans to add more commands?
Yes, future work like teaching the kernel how to cache device evidence and re-challenge a device after error or power-loss recovery [1]. It may even supplant some sysfs interfaces that would be better with transactional semantics.
For example, a LOCK operation that returns a session cookie and a RUN/ACCEPT operation that only succeeds if the session has not been invalidated in the interim. sysfs would require userspace locking for such a semantic.
[1]: http://lore.kernel.org/[email protected]
Quoted 1 line
The read-only stuff feels like it could be a sysfs API?
In fact, the original genesis of a proposal in this space was sysfs back at Plumbers 2024 [2].
As the number of attributes, modifiers, and transactions grew the feedback in the BoF was to move to a more suitable uAPI, netlink.
Yes, a subset of the objects here could move to sysfs [3], but that does relieve the main need here which is an interface that can dump a fresh copy of the device measurements (settings and device data up to 16MB in size), signed by the device, with a nonce provided by relying party (userspace).
[2]: https://lpc.events/event/18/contributions/1955/
[3]: http://lore.kernel.org/[email protected]
Quoted 2 lines
The main strength of Netlink is "do" commands with multiple optional attrs.
Yes, that is attractive and saves a pile of bug prone ioctl handling.
The gap I need to fill first though is a uAPI that allows for large blobs to be fetched after being regenerated / reformatted besed on some input attributes.
"Multi message netlink attributes" while inventive, feels less awkward and more future proof than a sysfs binary attribute scheme to do the same.
Jakub Kicinski
maintainerRe: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
Quoted 12 lines · On Mon, 16 Mar 2026 18:45:24 -0700 Dan Williams wrote:
Quoted 11 lines · Jakub Kicinski
Quoted 6 lines · On Mon, 2 Mar 2026 16:01:56 -0800 Dan Williams wrote:
The implementation adheres to the guideline from:
Documentation/userspace-api/netlink/genetlink-legacy.rstNew Netlink families should never respond to a DO operation with multiple replies, with ``NLM_F_MULTI`` set. Use a filtered dump instead.
My understanding of
F_MULTIis that deserializer is supposed to continue deserializing into current object. IOW if we have:
Quoted 7 lines
Quoted 5 lines
struct does_this { int really; int have_to; int be_netlink; };Heh, sensing a subtle message here...
One learns to optimize for conserving one's own attention wherever possible :>
Quoted 5 lines
Quoted 2 lines
The main strength of Netlink is "do" commands with multiple optional attrs.
Yes, that is attractive and saves a pile of bug prone ioctl handling.
Quoted 7 lines
The gap I need to fill first though is a uAPI that allows for large blobs to be fetched after being regenerated / reformatted besed on some input attributes.
"Multi message netlink attributes" while inventive, feels less awkward and more future proof than a sysfs binary attribute scheme to do the same.
Alright, so to make this more Netlink-y you can either:
- delete the F_MULTI and replicate other attrs in each message and add an offset attr; this will make each message in the dump more standalone.
- keep the F_MULTI but object_val has to be a multi-attr, and then we have to teach YNL to correctly append the attrs.
Former is definitely less work. Latter could end up being cleaner but there are some unknowns so hard to tell for sure; more plumbing.
Re: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
Jakub Kicinski wrote:
[..]
Quoted 20 lines
Quoted 5 lines
Quoted 2 lines
The main strength of Netlink is "do" commands with multiple optional attrs.
Yes, that is attractive and saves a pile of bug prone ioctl handling.
Quoted 7 lines
The gap I need to fill first though is a uAPI that allows for large blobs to be fetched after being regenerated / reformatted besed on some input attributes.
"Multi message netlink attributes" while inventive, feels less awkward and more future proof than a sysfs binary attribute scheme to do the same.
Alright, so to make this more Netlink-y you can either:
- delete theF_MULTIand replicate other attrs in each message and add an offset attr; this will make each message in the dump more standalone.
- keep theF_MULTIbutobject_valhas to be a multi-attr, and then we have to teach YNL to correctly append the attrs.
Quoted 2 lines
Former is definitely less work. Latter could end up being cleaner but there are some unknowns so hard to tell for sure; more plumbing.
Makes sense, and not yet sure which one is lower maintenance burden long term. I will play with it a bit.
I am leaning to the first option more for the fact that it puts the burden of being strange squarely on the implementation that wants it. If this discussion attracts more large blob dumpers then the second option.
Thanks for spending some attention here.
Lukas Wunner
reviewerRe: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
In reply to Jakub KicinskiQuoted 10 lines · Jakub Kicinski
Quoted 6 lines · On Mon, 2 Mar 2026 16:01:56 -0800 Dan Williams wrote:
The implementation adheres to the guideline from:
Documentation/userspace-api/netlink/genetlink-legacy.rstNew Netlink families should never respond to a DO operation with multiple replies, with ``NLM_F_MULTI`` set. Use a filtered dump instead.
My understanding of
F_MULTIis that deserializer is supposed to continue deserializing into current object.
So is the "should" above meant to be understood in the RFC 2119 way, i.e. as a mere recommendation?
The problem we're facing is that nlattr::nla_len is u16, so the maximum size is 65531 bytes (65535 minus header). That's insufficient for transmitting blobs that are several megabytes in size.
The obvious solution is to split the blobs into smaller chunks and transmit each chunk in an attribute of the same type. The application then concatenates them together to reconstruct the blob. For particularly large blobs, it may even be necessary to split across multiple messages by way of NLM_F_MULTI.
Apart from the attribute size limitation, there's the problem that copying large blobs in memory is inefficient. Ideally we'd want zero-copy. The solution I came up with is to attach the blob's pages as fragments to the skb. Conceptually the fragments succeed the linear buffer of the skb, so by putting the nlattr header into the linear buffer and attaching the blob as fragments, the receiver consumes the netlink message in a natural way. This patch introduces an nla_put_blob() helper which was pretty straightforward:
https://github.com/l1k/linux/commit/af9b939fc30b
This patch is taking advantage of the helper:
https://github.com/l1k/linux/commit/009663bd172e
The only change I had to make is amending nlmsg_end() to take the fragments into account when calculating the nlmsg_len.
The patch does achieve zero-copy on the sender's end. It may also achieve zero-copy on the receiver's end if the receiver is in the kernel. However it does *not* achieve zero-copy if the receiver is in user space. That's because:
simple_copy_to_iter()
copy_to_iter()
_copy_to_iter()
copy_to_user_iter()
raw_copy_to_user()... will just stupidly copy the data into the user space buffer. It might be possible to achieve zero-copy in user space via io_uring.
At this point perhaps your conclusion is that netlink isn't the right protocol for this job. It's great for transmitting sets of small items, some of which may be optional, but it's obviously not well-suited for large items.
Jason Gunthorpe was quite insistent that we use netlink and you know how consensus-oriented kernel development is. Indeed sysfs has turned out not to be ideal because the protocol that we're dealing with (SPDM - DMTF DSP0274) allows many degrees of freedom and making them available through sysfs quickly becomes unwieldy.
E.g. when installing a certificate onto a device, the protocol allows specifying additional parameters (a keypair ID and a certificate model) together with the certificate chain that shall be installed. That doesn't square well with the "one value per file" sysfs model. User space would have to write the keypair ID and certificate model to separate attributes, then write the certificate chain to a third attribute. So the kernel would need some kind of state machine to keep track of which sysfs attributes have been written. It gets quite ugly.
As another example, the SPDM protocol allows retrieving measurements from the device. The measurements are indexed by an 8-bit number. To expose them via sysfs, the kernel would have to retrieve all of them on device enumeration so that it knows which indices are populated and need to be exposed in sysfs. That would incur a delay on device enumeration and thus lead to slower boot times.
If netlink is at all the right protocol for the job, I'm wondering if an extension for larger attributes would be entertained. Basically a variation of struct nlattr, but with a 24-bit or 32-bit size and maybe a list of fragment numbers. The latter would be useful to have *multiple* zero-copy attributes because the patches linked above only allow for a single zero-copy attribute per nlmsg.
Thanks,
Lukas
Re: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
Lukas Wunner wrote:
[..]
Quoted 4 lines
At this point perhaps your conclusion is that netlink isn't the right protocol for this job. It's great for transmitting sets of small items, some of which may be optional, but it's obviously not well-suited for large items.
Right, and sysfs is not well suited for transaction in/out semantics.
Quoted 1 line
Jason Gunthorpe was quite insistent that we use netlink and you know
Jason can of course correct me, but the insistence was less that netlink was the right tool for the job, and more that sysfs was the wrong tool for the job.
Netlink appears to be the least worst option.
Quoted 4 lines
how consensus-oriented kernel development is. Indeed sysfs has turned out not to be ideal because the protocol that we're dealing with (SPDM - DMTF DSP0274) allows many degrees of freedom and making them available through sysfs quickly becomes unwieldy.
[..]
Quoted 2 lines
If netlink is at all the right protocol for the job, I'm wondering if an extension for larger attributes would be entertained.
It is still not clear to me that allowing larger attributes are part of the solution space. They seem to be a premature performance optimization once userspace is prepared to reassemble a large blob over multiple messages.
Jason Gunthorpe
maintainerRe: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
Quoted 7 lines · Dan Williams
Lukas Wunner wrote:
[..]Quoted 4 lines
At this point perhaps your conclusion is that netlink isn't the right protocol for this job. It's great for transmitting sets of small items, some of which may be optional, but it's obviously not well-suited for large items.
Quoted 7 lines
Right, and sysfs is not well suited for transaction in/out semantics.
Quoted 1 line
Jason Gunthorpe was quite insistent that we use netlink and you know
Jason can of course correct me, but the insistence was less that netlink was the right tool for the job, and more that sysfs was the wrong tool for the job.
+1
netlink is a good starting point, but if it isn't fitting well then the next stop would be an ioctl char dev..
Jason
Lukas Wunner
reviewerRe: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
In reply to Dan WilliamsQuoted 5 lines · Dan Williams
Documentation/netlink/specs/pci-tsm.yaml new file +5
-0+definitions: + - + type: const + name: max-object-size + value: 0x01000000
[...]
Documentation/netlink/specs/pci-tsm.yaml line 130
Quoted 5 lines
Documentation/netlink/specs/pci-tsm.yaml new file +5
-0+ - + name: val + type: binary + checks: + max-len: max-obj-size
The length of a netlink attribute is a 16-bit value, so a 16 MByte value (0x01000000) won't fit.
Moreover you're referencing max-obj-size but are defining max-object-size.
This doesn't look like it's ever been tested, so at the very least it should be marked RFC in the subject to convey that it's not yet in a cut-and-dried state.
The two top-most commits on my development branch have solved the size problem and may serve as a template:
https://github.com/l1k/linux/commits/doe
Thanks,
Lukas
Re: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
Quoted 13 lines · Lukas Wunner
Quoted 5 lines · Dan Williams
Documentation/netlink/specs/pci-tsm.yaml new file +5
-0+definitions: + - + type: const + name: max-object-size + value: 0x01000000[...]
Documentation/netlink/specs/pci-tsm.yaml line 130
Quoted 5 lines
Documentation/netlink/specs/pci-tsm.yaml new file +5
-0+ - + name: val + type: binary + checks: + max-len: max-obj-size
Quoted 4 lines
The length of a netlink attribute is a 16-bit value, so a 16 MByte value (
0x01000000) won't fit.Moreover you're referencing max-obj-size but are defining max-object-size.
Good catch, not sure why the tooling did not complain.
Quoted 3 lines
This doesn't look like it's ever been tested, so at the very least it should be marked RFC in the subject to convey that it's not yet in a cut-and-dried state.
The 16MB limit has indeed not been tested, the test script in this set was using smaller than 64K payloads to check out the interface.
The RFC comment for this piece is fair. The whole became a v2 based on the maturity of other proposals that were in v1. This "evidence" proposal deserves its own conversation.
Quoted 4 lines
The two top-most commits on my development branch have solved the size problem and may serve as a template:
I was concerned that gets the same "too inventive" feedback from netdev folks, but I ended up triggering the same with a broken alternate proposal.
Re: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
Quoted 14 lines · Dan Williams
Quoted 13 lines · Lukas Wunner
Quoted 5 lines · Dan Williams
Documentation/netlink/specs/pci-tsm.yaml new file +5
-0+definitions: + - + type: const + name: max-object-size + value: 0x01000000[...]
Quoted 5 lines
Documentation/netlink/specs/pci-tsm.yaml new file +5
-0+ - + name: val + type: binary + checks: + max-len: max-obj-size
Quoted 6 lines
Quoted 4 lines
The length of a netlink attribute is a 16-bit value, so a 16 MByte value (
0x01000000) won't fit.Moreover you're referencing max-obj-size but are defining max-object-size.
Good catch, not sure why the tooling did not complain.
This ends up not being a problem because 'val' is not referenced by a request, only a reply. The automatic code generation does not generate an nla_policy for replies. It should just be deleted because, see below...
Quoted 6 lines
Quoted 3 lines
This doesn't look like it's ever been tested, so at the very least it should be marked RFC in the subject to convey that it's not yet in a cut-and-dried state.
The 16MB limit has indeed not been tested, the test script in this set was using smaller than 64K payloads to check out the interface.
So 16MB works ok, slow, but works. A given attribute in this implementation never exceeds the limit because the protocol is for assembly of a bulk attribute over multiple messages.
Quoted 4 lines
The RFC comment for this piece is fair. The whole became a v2 based on the maturity of other proposals that were in v1. This "evidence" proposal deserves its own conversation.
Quoted 8 lines
Quoted 4 lines
The two top-most commits on my development branch have solved the size problem and may serve as a template:
I was concerned that gets the same "too inventive" feedback from netdev folks, but I ended up triggering the same with a broken alternate proposal.
I think your patches address the performance problem if userspace passes large recieve buffers, but I do not think we need to make that a userspace requirement.
Lukas Wunner
reviewerRe: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
Quoted 11 lines · Dan Williams
Quoted 7 lines · Dan Williams
Quoted 3 lines · Lukas Wunner
This doesn't look like it's ever been tested, so at the very least it should be marked RFC in the subject to convey that it's not yet in a cut-and-dried state.
The 16MB limit has indeed not been tested, the test script in this set was using smaller than 64K payloads to check out the interface.
So 16MB works ok, slow, but works. A given attribute in this implementation never exceeds the limit
Famous last words.
If you look at netlink_dump(), it sizes the skb based on nlk->max_recvmsg_len. If that's larger than 64k, you'll try to fill as much as possible of that space with a single netlink attribute. The computation of "available" in your patch doesn't take the 65531 bytes limit for a netlink attribute into account so it looks like you'll end up overflowing the length of the netlink attribute.
Unfortunately nla_put() doesn't prevent such overflows, it does all the size calculations with an int, not a u16.
Thanks,
Lukas
Re: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
Quoted 7 lines · Lukas Wunner
Quoted 6 lines · Dan Williams
Quoted 5 lines · Dan Williams
Quoted 3 lines · Lukas Wunner
This doesn't look like it's ever been tested, so at the very least it should be marked RFC in the subject to convey that it's not yet in a cut-and-dried state.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
The 16MB limit has indeed not been tested, the test script in this set was using smaller than 64K payloads to check out the interface.
So 16MB works ok, slow, but works. A given attribute in this implementation never exceeds the limit
Famous last words.
I am not convinced you have found the "gotcha" you think you have...
Quoted 7 lines
If you look at
netlink_dump(), it sizes the skb based onnlk->max_recvmsg_len. If that's larger than 64k, you'll try to fill as much as possible of that space with a single netlink attribute. The computation of "available" in your patch doesn't take the 65531 bytes limit for a netlink attribute into account so it looks like you'll end up overflowing the length of the netlink attribute.
The @len to nla_put() should not overflow because it is based on the available tailroom in the skb minus netlink overhead. The "inventive" hack that Jakub is reacting to is that this scheme requires a receiver that assumes repeating the attribute in the receive stream must be handled as concatenation.
Quoted 2 lines
Unfortunately
nla_put()doesn't prevent such overflows, it does all the size calculations with an int, not a u16.
Are we looking at the same capacity calculation?
len = min(available - overhead, object_len - ctx->offset);
Lukas Wunner
reviewerRe: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
In reply to Dan WilliamsQuoted 3 lines · Dan Williams
Documentation/netlink/specs/pci-tsm.yaml new file +3
-0+ type: const + name: max-nonce-size + value: 256
[...]
include/uapi/linux/pci-tsm-netlink.h line 15
Quoted 3 lines
include/uapi/linux/pci-tsm-netlink.h new file +3
-0+#define PCI_TSM_MAX_OBJECT_SIZE 16777216 +#define PCI_TSM_MAX_NONCE_SIZE 256 +#define PCI_TSM_MAX_OBJ_TYPE 4
Where is the maximum nonce size of 256 bytes coming from?
Such definitions should always be accompanied by a spec reference, not pulled out of thin air.
SPDM nonces are 32 bytes, I assume that's what we're dealing with here?
This patch:
https://github.com/l1k/linux/commit/bca645e08ee9
... contains the following definition:
#define SPDM_NONCE_SZ 32 /* SPDM 1.0.0 table 20 */
Though it's defined in a private header in lib/spdm/spdm.h. If there's a need outside of the SPDM library, its visibility can be broadened of course.
Thanks,
Lukas
Re: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
Quoted 10 lines · Lukas Wunner
Quoted 3 lines · Dan Williams
Documentation/netlink/specs/pci-tsm.yaml new file +3
-0+ type: const + name: max-nonce-size + value: 256[...]
include/uapi/linux/pci-tsm-netlink.h line 15
Quoted 3 lines
include/uapi/linux/pci-tsm-netlink.h new file +3
-0+#define PCI_TSM_MAX_OBJECT_SIZE 16777216 +#define PCI_TSM_MAX_NONCE_SIZE 256 +#define PCI_TSM_MAX_OBJ_TYPE 4Where is the maximum nonce size of 256 bytes coming from?
I took it from Aneesh's off-list RFC, and meant to circle back with him. Yes, it should come with a spec reference.
I am having trouble finding a clear reference for ARM CCA that clarifies that measurement recollection takes the SPDM standard nonce as an input. Perhaps the document I have "DEN0137 1.1-alp8" is out of date? TDX and SEV-TIO do reference SPDM for the nonce size.
Quoted 2 lines
Such definitions should always be accompanied by a spec reference, not pulled out of thin air.
Yes.
Quoted 11 lines
SPDM nonces are 32 bytes, I assume that's what we're dealing with here?
This patch:
https://github.com/l1k/linux/commit/bca645e08ee9... contains the following definition:
#defineSPDM_NONCE_SZ32 /* SPDM 1.0.0 table 20 */Though it's defined in a private header in
lib/spdm/spdm.h. If there's a need outside of the SPDM library, its visibility can be broadened of course.
Of course. Again this points to a need to pull this proposal out separate from the rest.
The ARM CCA spec does reference the EAT nonce which is 64-bytes. So it may be the case that PCI_TSM_MAX_NONCE_SIZE != SPDM_NONCE_SZ depending on what evidence can be collected over this interface, but I am not finding any spec references for 256, Aneesh?
Aneesh Kumar K.V
maintainerRe: [PATCH v2 08/19] PCI/TSM: Add "evidence" support
Quoted 4 lines · Dan Williams
The ARM CCA spec does reference the EAT nonce which is 64-bytes. So it may be the case that
PCI_TSM_MAX_NONCE_SIZE!=SPDM_NONCE_SZdepending on what evidence can be collected over this interface, but I am not finding any spec references for 256, Aneesh?
That was probably me getting confused about 256 bits in an earlier version of the code. Also, the RmiVdevMeasureParams structure layout adds 256 bytes of padding between the different fields.
-aneesh
PATCH v2 9/19 PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
In reply to Dan WilliamsAfter pci_tsm_bind() and pci_tsm_lock() the low level TSM driver is expected to populate PCI_TSM_EVIDENCE_TYPE_REPORT in its evidence store. This report is defined by the TDISP GET_DEVICE_INTERFACE_REPORT response payload.
Add a helper to create encrypted MMIO descriptors from that report data. With those descriptors the TSM driver can use pci_tsm_mmio_setup() to inform ioremap() how to map the device per the device's expectations. The VM is expected to validate the interface with the relying party before accepting the device for operation.
The helper also provides the obfuscated starting address for each encrypted MMIO range as the VM is never disclosed on the hpa that correlates to the gpa of the device's mmio. The obfuscated address is BAR relative.
Based on an original patch by Aneesh [1]
--- include/linux/ioport.h | 1 + include/linux/pci-tsm.h | 34 ++++++ drivers/pci/tsm/core.c | 235 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 270 insertions(+)
include/linux/ioport.h +1 -0
@@ -143,6 +143,7 @@ enum {IORES_DESC_RESERVED =7 ,IORES_DESC_SOFT_RESERVED =8 ,IORES_DESC_CXL =9 , + IORES_DESC_ENCRYPTED = 10, };/*
include/linux/pci-tsm.h +34 -0
@@ -194,12 +194,42 @@ struct pci_tsm_pf0 {struct pci_doe_mb *doe_mb; }; +/** +* struct pci_tsm_mmio_entry - an encrypted MMIO range +* @res: MMIO address range (typically Guest Physical Address, GPA) +* @tsm_offset: Host Physical Address, HPA obfuscation offset added by the TSM. +* Translates report addresses to GPA. +*/ +struct pci_tsm_mmio_entry { +struct resource res; +u64 tsm_offset; +}; + +struct pci_tsm_mmio { +int nr; + struct pci_tsm_mmio_entry mmio[] __counted_by(nr); +}; + +static inline struct pci_tsm_mmio_entry * +pci_tsm_mmio_entry (struct pci_tsm_mmio *mmio,int idx) +{ +return &mmio->mmio[idx]; +} + +static inline struct resource *pci_tsm_mmio_resource (struct pci_tsm_mmio *mmio, +int idx) +{ +return &mmio->mmio[idx].res; +} +/** * struct pci_tsm_devsec - context for tracking private/accepted PCI resources * @base_tsm: generic core "tsm" context +* @mmio: encrypted MMIO resources for this assigned device */ struct pci_tsm_devsec {struct pci_tsm base_tsm; +struct pci_tsm_mmio *mmio; };/* physical function0 and capable of 'connect' */
@@ -297,6 +327,10 @@ ssize_t pci_tsm_guest_req(struct pci_dev *pdev, enum pci_tsm_req_scope scope,struct pci_tsm_devsec *to_pci_tsm_devsec (struct pci_tsm *tsm);void pci_tsm_init_evidence (struct pci_tsm_evidence *evidence,int slot,enum hash_algo digest_algo); +int pci_tsm_mmio_setup (struct pci_dev *pdev,struct pci_tsm_mmio *mmio); +void pci_tsm_mmio_teardown (struct pci_tsm_mmio *mmio); +struct pci_tsm_mmio *pci_tsm_mmio_alloc (struct pci_dev *pdev); +int pci_tsm_mmio_free (struct pci_dev *pdev,struct pci_tsm_mmio *mmio);#else static inline int pci_tsm_register (struct tsm_dev *tsm_dev) {
drivers/pci/tsm/core.c +235 -0
247 lines not on this page · raw message
Signature · 1 line
2.52.0
Re: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 23 lines · Dan Williams
After
pci_tsm_bind()andpci_tsm_lock()the low level TSM driver is expected to populatePCI_TSM_EVIDENCE_TYPE_REPORTin its evidence store. This report is defined by the TDISPGET_DEVICE_INTERFACE_REPORTresponse payload.Add a helper to create encrypted MMIO descriptors from that report data. With those descriptors the TSM driver can use
pci_tsm_mmio_setup()to informioremap()how to map the device per the device's expectations. The VM is expected to validate the interface with the relying party before accepting the device for operation.The helper also provides the obfuscated starting address for each encrypted MMIO range as the VM is never disclosed on the hpa that correlates to the gpa of the device's mmio. The obfuscated address is BAR relative.
Based on an original patch by Aneesh [1]
Cc Arnd Bergmann <[email protected]>Co-developed-by Xu Yilun <[email protected]>Signed-off-by Xu Yilun <[email protected]>Signed-off-by Dan Williams <[email protected]>
[..]
Quoted 20 lines
drivers/pci/tsm/core.c +20
-0+/** +* pci_tsm_mmio_alloc() - allocate encrypted MMIO range descriptor +* @pdev: device owner of MMIO ranges +* @report_data: TDISP Device Interface (DevIf) Report blob +* @report_sz: DevIf Report size +* +* Return: the encrypted MMIO range descriptor on success, NULL on failure +* +* Assumes that this is called within the live lifetime of a PCI device's +* association with a low level TSM. +*/ +struct pci_tsm_mmio *pci_tsm_mmio_alloc (struct pci_dev *pdev) +{ +struct pci_tsm *tsm = pdev->tsm; +struct pci_tsm_evidence *evidence = &tsm->evidence; +struct pci_tsm_evidence_object *report_obj = &evidence->obj[PCI_TSM_EVIDENCE_TYPE_REPORT ]; +struct tsm_dev *tsm_dev = tsm->tsm_dev; +u64 reporting_bar_base, last_reporting_end; +const struct pci_tsm_devif_report *report; +u32 mmio_range_count;
drivers/pci/tsm/core.c static ssize_t dsm_show(struct device *dev, struct device_attribute *attr, · line 679
Quoted 10 lines
drivers/pci/tsm/core.c +10
-0+int last_bar =-1 ; +int i; + +guard (rwsem_read)(&evidence->lock); +if (report_obj->len <sizeof (struct pci_tsm_devif_report )) +return NULL ; + +if (dev_WARN_ONCE (&tsm_dev->dev, !IS_ALIGNED ((unsigned long ) report_obj->data,8 ), +"misaligned report data\n" )) +return NULL ;
Is this going to cause any implementation to need to copy the buffer received from the low-level TSM? If so I would just mark 'struct pci_tsm_devif_report' and 'struct pci_tsm_mmio_entry' as __packed and drop this check.
Quoted 19 lines
drivers/pci/tsm/core.c +19
-0+ + report = report_obj->data; + mmio_range_count =__le32_to_cpu (report->mmio_range_count); + +/* check that the report object is self-consistent on mmio entries */ +if (report_obj->len <struct_size (report, mmio, mmio_range_count)) +return NULL ; + +/* create pci_tsm_mmio descriptors from the report data */ + struct pci_tsm_mmio *mmio __free(kfree) = +kzalloc (struct_size (mmio, mmio, mmio_range_count),GFP_KERNEL ); +if (!mmio) +return NULL ; + +for (i =0 ; i < mmio_range_count; i++) { +u64 range_off; +struct range range; +const struct pci_tsm_devif_mmio *mmio_data = &report->mmio[i]; + struct pci_tsm_mmio_entry *entry =
drivers/pci/tsm/core.c static ssize_t dsm_show(struct device *dev, struct device_attribute *attr, · line 708
Quoted 10 lines
drivers/pci/tsm/core.c +10
-0+pci_tsm_mmio_entry (mmio, mmio->nr); +/* report values in are in terms of 4K pages */ +u64 tsm_offset =__le64_to_cpu (mmio_data->pfn) *SZ_4K ; +u64 size =__le32_to_cpu (mmio_data->nr_pfns) *SZ_4K ; +u32 attr =__le32_to_cpu (mmio_data->attributes); +int bar =FIELD_GET (PCI_TSM_DEVIF_REPORT_MMIO_ATTR_RANGE_ID , + attr); + + tsm_offset *=SZ_4K ; + size *=SZ_4K ;
Whoops, these pfn to absolute address conversions were already performed above, will fix.
Re: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 6 lines
drivers/pci/tsm/core.c static ssize_t dsm_show(struct device *dev, struct device_attribute *attr, · line 679
Quoted 3 lines
drivers/pci/tsm/core.c +3
-0+if (dev_WARN_ONCE (&tsm_dev->dev, !IS_ALIGNED ((unsigned long ) report_obj->data,8 ), +"misaligned report data\n" )) +return NULL ;Is this going to cause any implementation to need to copy the buffer received from the low-level TSM? If so I would just mark 'struct
TDX Connect needs the copy, the GHCI header is 20 byte size, not naturally aligned...
Quoted 2 lines
pci_tsm_devif_report' and 'structpci_tsm_mmio_entry' as__packedand drop this check.
..But since this series will persist the report blob long term in tsm.evidence.obj[PCI_TSM_EVIDENCE_TYPE_REPORT], we anyway need to copy the blob out from temporary shared GHCI communication buffer. So it is actually good to me.
Aneesh Kumar K.V
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
In reply to Dan WilliamsQuoted 287 lines · Dan Williams
After
pci_tsm_bind()andpci_tsm_lock()the low level TSM driver is expected to populatePCI_TSM_EVIDENCE_TYPE_REPORTin its evidence store. This report is defined by the TDISPGET_DEVICE_INTERFACE_REPORTresponse payload.Add a helper to create encrypted MMIO descriptors from that report data. With those descriptors the TSM driver can use
pci_tsm_mmio_setup()to informioremap()how to map the device per the device's expectations. The VM is expected to validate the interface with the relying party before accepting the device for operation.The helper also provides the obfuscated starting address for each encrypted MMIO range as the VM is never disclosed on the hpa that correlates to the gpa of the device's mmio. The obfuscated address is BAR relative.
Based on an original patch by Aneesh [1]
Cc Arnd Bergmann <[email protected]>Co-developed-by Xu Yilun <[email protected]>Signed-off-by Xu Yilun <[email protected]>Signed-off-by Dan Williams <[email protected]>--- include/linux/ioport.h | 1 + include/linux/pci-tsm.h | 34 ++++++ drivers/pci/tsm/core.c | 235 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 270 insertions(+)include/linux/ioport.h +1
-0@@ -143,6 +143,7 @@ enum {IORES_DESC_RESERVED =7 ,IORES_DESC_SOFT_RESERVED =8 ,IORES_DESC_CXL =9 , + IORES_DESC_ENCRYPTED = 10, };/* include/linux/pci-tsm.h +34
-0@@ -194,12 +194,42 @@ struct pci_tsm_pf0 {struct pci_doe_mb *doe_mb; }; +/** +* struct pci_tsm_mmio_entry - an encrypted MMIO range +* @res: MMIO address range (typically Guest Physical Address, GPA) +* @tsm_offset: Host Physical Address, HPA obfuscation offset added by the TSM. +* Translates report addresses to GPA. +*/ +struct pci_tsm_mmio_entry { +struct resource res; +u64 tsm_offset; +}; + +struct pci_tsm_mmio { +int nr; + struct pci_tsm_mmio_entry mmio[] __counted_by(nr); +}; + +static inline struct pci_tsm_mmio_entry * +pci_tsm_mmio_entry (struct pci_tsm_mmio *mmio,int idx) +{ +return &mmio->mmio[idx]; +} + +static inline struct resource *pci_tsm_mmio_resource (struct pci_tsm_mmio *mmio, +int idx) +{ +return &mmio->mmio[idx].res; +} +/** * struct pci_tsm_devsec - context for tracking private/accepted PCI resources * @base_tsm: generic core "tsm" context +* @mmio: encrypted MMIO resources for this assigned device */ struct pci_tsm_devsec {struct pci_tsm base_tsm; +struct pci_tsm_mmio *mmio; };/* physical function0 and capable of 'connect' */ @@ -297,6 +327,10 @@ ssize_t pci_tsm_guest_req(struct pci_dev *pdev, enum pci_tsm_req_scope scope,struct pci_tsm_devsec *to_pci_tsm_devsec (struct pci_tsm *tsm);void pci_tsm_init_evidence (struct pci_tsm_evidence *evidence,int slot,enum hash_algo digest_algo); +int pci_tsm_mmio_setup (struct pci_dev *pdev,struct pci_tsm_mmio *mmio); +void pci_tsm_mmio_teardown (struct pci_tsm_mmio *mmio); +struct pci_tsm_mmio *pci_tsm_mmio_alloc (struct pci_dev *pdev); +int pci_tsm_mmio_free (struct pci_dev *pdev,struct pci_tsm_mmio *mmio);#else static inline int pci_tsm_register (struct tsm_dev *tsm_dev) {drivers/pci/tsm/core.c +172
-0@@ -15,6 +15,7 @@#include <linux/pci-tsm.h> #include <linux/sysfs.h> #include <linux/tsm.h> +#include <linux/unaligned.h> #include <linux/xarray.h> #include "../pci.h" @@ -558,6 +559,240 @@ static ssize_t dsm_show(struct device *dev, struct device_attribute *attr, }static DEVICE_ATTR_RO (dsm); +static void mmio_teardown (struct pci_tsm_mmio *mmio,int nr) +{ +while (nr--) +remove_resource (pci_tsm_mmio_resource (mmio, nr)); +} + +/** +* pci_tsm_mmio_setup() - mark device MMIO as encrypted in iomem +* @pdev: device owner of MMIO resources +* @mmio: container of an array of resources to mark encrypted +*/ +int pci_tsm_mmio_setup (struct pci_dev *pdev,struct pci_tsm_mmio *mmio) +{ +int i; + +device_lock_assert (&pdev->dev); +if (pdev->dev.driver) +return -EBUSY ; + +for (i =0 ; i < mmio->nr; i++) { +struct resource *res =pci_tsm_mmio_resource (mmio, i); +int j; + +if (resource_size (res) ==0 || !res->end) +break ; + +/* Only require the caller to set the range, init remainder */ + *res =DEFINE_RES_NAMED_DESC (res->start,resource_size (res), +"PCI MMIO Encrypted" , +IORESOURCE_MEM , +IORES_DESC_ENCRYPTED ); + +for (j =0 ; j <PCI_NUM_RESOURCES ; j++) +if (resource_contains (pci_resource_n (pdev, j), res)) +break ; + +/* Request is outside of device MMIO */ +if (j >=PCI_NUM_RESOURCES ) +break ; + +if (insert_resource (&iomem_resource, res) !=0 ) +break ; + } + +if (i >= mmio->nr) +return 0 ; + +mmio_teardown (mmio, i); + +return -EINVAL ; +} +EXPORT_SYMBOL_GPL (pci_tsm_mmio_setup); + +void pci_tsm_mmio_teardown (struct pci_tsm_mmio *mmio) +{ +mmio_teardown (mmio, mmio->nr); +} +EXPORT_SYMBOL_GPL (pci_tsm_mmio_teardown); + +/* +* PCIe ECN TEE Device Interface Security Protocol (TDISP) +* +* Device Interface Report data object layout as defined by PCIe r7.0 section +* 11.3.11 +*/ +#define PCI_TSM_DEVIF_REPORT_MMIO_ATTR_MSIX_TABLE BIT(0) +#define PCI_TSM_DEVIF_REPORT_MMIO_ATTR_MSIX_PBA BIT(1) +#define PCI_TSM_DEVIF_REPORT_MMIO_ATTR_IS_NON_TEE BIT(2) +#define PCI_TSM_DEVIF_REPORT_MMIO_ATTR_IS_UPDATABLE BIT(3) +#define PCI_TSM_DEVIF_REPORT_MMIO_ATTR_RANGE_ID GENMASK(31, 16) + +/* An interface report 'pfn' is 4K in size */ +struct pci_tsm_devif_mmio { +__le64 pfn; +__le32 nr_pfns; +__le32 attributes; +}; + +struct pci_tsm_devif_report { +__le16 interface_info; +__le16 reserved; +__le16 msi_x_message_control; +__le16 lnr_control; +__le32 tph_control; +__le32 mmio_range_count; +struct pci_tsm_devif_mmio mmio[]; +}; + +/** +* pci_tsm_mmio_alloc() - allocate encrypted MMIO range descriptor +* @pdev: device owner of MMIO ranges +* @report_data: TDISP Device Interface (DevIf) Report blob +* @report_sz: DevIf Report size +* +* Return: the encrypted MMIO range descriptor on success, NULL on failure +* +* Assumes that this is called within the live lifetime of a PCI device's +* association with a low level TSM. +*/ +struct pci_tsm_mmio *pci_tsm_mmio_alloc (struct pci_dev *pdev) +{ +struct pci_tsm *tsm = pdev->tsm; +struct pci_tsm_evidence *evidence = &tsm->evidence; +struct pci_tsm_evidence_object *report_obj = &evidence->obj[PCI_TSM_EVIDENCE_TYPE_REPORT ]; +struct tsm_dev *tsm_dev = tsm->tsm_dev; +u64 reporting_bar_base, last_reporting_end; +const struct pci_tsm_devif_report *report; +u32 mmio_range_count; +int last_bar =-1 ; +int i; + +guard (rwsem_read)(&evidence->lock); +if (report_obj->len <sizeof (struct pci_tsm_devif_report )) +return NULL ; + +if (dev_WARN_ONCE (&tsm_dev->dev, !IS_ALIGNED ((unsigned long ) report_obj->data,8 ), +"misaligned report data\n" )) +return NULL ; + + report = report_obj->data; + mmio_range_count =__le32_to_cpu (report->mmio_range_count); + +/* check that the report object is self-consistent on mmio entries */ +if (report_obj->len <struct_size (report, mmio, mmio_range_count)) +return NULL ; + +/* create pci_tsm_mmio descriptors from the report data */ + struct pci_tsm_mmio *mmio __free(kfree) = +kzalloc (struct_size (mmio, mmio, mmio_range_count),GFP_KERNEL ); +if (!mmio) +return NULL ; + +for (i =0 ; i < mmio_range_count; i++) { +u64 range_off; +struct range range; +const struct pci_tsm_devif_mmio *mmio_data = &report->mmio[i]; +struct pci_tsm_mmio_entry *entry = +pci_tsm_mmio_entry (mmio, mmio->nr); +/* report values in are in terms of 4K pages */ +u64 tsm_offset =__le64_to_cpu (mmio_data->pfn) *SZ_4K ; +u64 size =__le32_to_cpu (mmio_data->nr_pfns) *SZ_4K ; +u32 attr =__le32_to_cpu (mmio_data->attributes); +int bar =FIELD_GET (PCI_TSM_DEVIF_REPORT_MMIO_ATTR_RANGE_ID , + attr); + + tsm_offset *=SZ_4K ; + size *=SZ_4K ; + +if (bar >=PCI_STD_NUM_BARS || + !(pci_resource_flags (pdev, bar) &IORESOURCE_MEM )) { +pci_dbg (pdev,"Invalid reporting bar ID %d\n" , bar); +return NULL ; + } + +if (last_bar > bar) { +pci_dbg (pdev,"Reporting bar ID not in ascending order\n" ); +return NULL ; + } + +if (last_bar < bar) { +/* transition to a new bar */ + last_bar = bar; +/* +* The tsm_offset for the first range of the BAR +* corresponds to the BAR base. +*/ + reporting_bar_base = tsm_offset; + }else if (tsm_offset < last_reporting_end) { +pci_dbg (pdev,"Reporting ranges within BAR not in ascending order\n" ); +return NULL ; + }
....
drivers/pci/tsm/core.c static ssize_t dsm_show(struct device *dev, struct device_attribute *attr, · line 740
Quoted 2 lines
drivers/pci/tsm/core.c +1
-0+ range_off = tsm_offset - reporting_bar_base;
range_off will always be zero? Should we do
range_off = tsm_offset & (pci_resource_len(pdev, bar) - 1);
So that we correctly handle if the interface report is reporting a range within a bar. The only requirement here is bar address should be aligned to its size and mmio_reporting_offset should not add offsets in that range.
Quoted 68 lines
drivers/pci/tsm/core.c +63
-0+ + last_reporting_end = tsm_offset + size; +if (last_reporting_end < tsm_offset) { +pci_dbg (pdev,"Reporting range overflow\n" ); +return NULL ; + } + + range_off = tsm_offset - reporting_bar_base; +if (pci_resource_len (pdev, bar) < range_off + size) { +pci_dbg (pdev,"Reporting range larger than BAR size\n" ); +return NULL ; + } + + range.start =pci_resource_start (pdev, bar) + range_off; + range.end = range.start + size -1 ; + +if (FIELD_GET (PCI_TSM_DEVIF_REPORT_MMIO_ATTR_IS_NON_TEE , + attr)) { +pci_dbg (pdev,"Skipping non-TEE range, BAR%d %pra\n" , + bar, &range); +continue ; + } + +/* Currently not supported */ +if (FIELD_GET (PCI_TSM_DEVIF_REPORT_MMIO_ATTR_MSIX_TABLE , + attr) || +FIELD_GET (PCI_TSM_DEVIF_REPORT_MMIO_ATTR_MSIX_PBA , attr)) { +pci_dbg (pdev,"Skipping MSIX range BAR%d %pra\n" , bar, + &range); +continue ; + } + + entry->res.start = range.start; + entry->res.end = range.end; + entry->tsm_offset = tsm_offset; + mmio->nr++; + } + +return_ptr (mmio); +} +EXPORT_SYMBOL_GPL (pci_tsm_mmio_alloc); + +/** +* pci_tsm_mmio_free() - free a pci_tsm_mmio instance +* @pdev: device owner of MMIO ranges +* @mmio: instance to free +* +* Returns 0 if @mmio was idle on entry, -EBUSY otherwise +*/ +int pci_tsm_mmio_free (struct pci_dev *pdev,struct pci_tsm_mmio *mmio) +{ +for (int i =0 ; i < mmio->nr; i++) { +struct resource *res =pci_tsm_mmio_resource (mmio, i); + +if (dev_WARN_ONCE (&pdev->dev,resource_assigned (res), +"MMIO resource still assigned %pr\n" , res)) +return -EBUSY ; + } +kfree (mmio); +return 0 ; +} +EXPORT_SYMBOL_GPL (pci_tsm_mmio_free); +/** * pci_tsm_accept() - accept a device for private MMIO+DMA operation * @pdev: PCI device to accept Signature · 1 line
2.52.0
Re: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 14 lines
Quoted 13 lines
drivers/pci/tsm/core.c +12
-0+if (last_bar < bar) { +/* transition to a new bar */ + last_bar = bar; +/* +* The tsm_offset for the first range of the BAR +* corresponds to the BAR base. +*/ + reporting_bar_base = tsm_offset; + }else if (tsm_offset < last_reporting_end) { +pci_dbg (pdev,"Reporting ranges within BAR not in ascending order\n" ); +return NULL ; + }....
Quoted 3 lines
drivers/pci/tsm/core.c static ssize_t dsm_show(struct device *dev, struct device_attribute *attr, · line 740
Quoted 2 lines
drivers/pci/tsm/core.c +1
-0+ range_off = tsm_offset - reporting_bar_base;
range_offwill always be zero? Should we do
tsm_offset comes from Device Interface Report, MMIO RANGE, First 4k Page. How do you interpret the exact meaning of this field?
My understanding is, it is the obfuscated host start pfn of this range, if this range has offset to the BAR start, this field should also be offsetted.
But if the first range in the BAR should be aligned to BAR, otherwise there is no way for guest to position the range in the BAR.
So the logic here is:
reporting_bar_base: the first obfuscated pfn for the BAR, the BAR pfn
tsm_offset: the current obfucated pfn for the BAR.
tsm_offset - reporting_bar_base: the offset to the BAR.
Quoted 7 lines
range_off=tsm_offset& (pci_resource_len(pdev, bar)- 1);So that we correctly handle if the interface report is reporting a range within a bar. The only requirement here is bar address should be aligned to its size and
mmio_reporting_offsetshould not add offsets in that range.
Jason Gunthorpe
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 3 lines · Xu Yilun
My understanding is, it is the obfuscated host start pfn of this range, if this range has offset to the BAR start, this field should also be offsetted.
The OS must get an idea of the bar layout out of the report, so there have to be restrictions on how it is formed otherwise it is unparsible. IMHO the PCI spec created this very general mechanism but the CPU CC specs need to constrain it to be usable by an OS.
Quoted 5 lines
Quoted 5 lines
range_off=tsm_offset& (pci_resource_len(pdev, bar)- 1);So that we correctly handle if the interface report is reporting a range within a bar. The only requirement here is bar address should be aligned to its size and
mmio_reporting_offsetshould not add offsets in that range.
Right.
Jason
Re: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Xu Yilun
My understanding is, it is the obfuscated host start pfn of this range, if this range has offset to the BAR start, this field should also be offsetted.
The OS must get an idea of the bar layout out of the report, so there have to be restrictions on how it is formed otherwise it is unparsible. IMHO the PCI spec created this very general mechanism but
Yeah, now I think I've fully understood the problem. The essential thing is some of the MMIO ranges could be hidden from the report, at least the MSIX/PBA ranges must be hidden if not locked/private. This makes guest hard to find the bar layout out of the report.
Quoted 7 lines
the CPU CC specs need to constrain it to be usable by an OS.
Quoted 5 lines
Quoted 5 lines
range_off=tsm_offset& (pci_resource_len(pdev, bar)- 1);So that we correctly handle if the interface report is reporting a range within a bar. The only requirement here is bar address should be aligned to its size and
mmio_reporting_offsetshould not add offsets in that range.
This is the solution on CPU CC side, which requires the TSM to choose a mmio_reporting_offset aligned to the MAX BAR size so that the obfuscation won't blur the bar offset info. This is a trade off, the larger the bar_size, the weaker the obfuscation. Is there a worst case, that a device has a large bar in high address and a small bar in low address, the small bar address has no intersection with the mmio_reporting_offset. Then the obfuscated address ends up being amusing 0xabcd012345000, which clearly leaks the mmio_reporting_offset and the physical BAR address...
And I've no idea why unlocked MSIX/PBA must be hidden? How about other non-TEE ranges, must be hidden or mustn't? Is there a possibility we enforce DSM to present all ranges, then the layout is clear to OS?
Quoted 4 lines
Right.
Jason
Re: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Xu Yilun wrote:
[..]
Quoted 3 lines
And I've no idea why unlocked MSIX/PBA must be hidden? How about other non-TEE ranges, must be hidden or mustn't? Is there a possibility we enforce DSM to present all ranges, then the layout is clear to OS?
Just to close this question, this was discussed at the last device security call. Indeed the expectation is that Linux will assume that all ranges besides MSIX/PBA must be present in the report, and that offset is always aligned. If / when an implementation violates that expectation they can help write the Linux quirk for that case or otherwise fix their implementation.
Jason Gunthorpe
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 6 lines · Dan Williams
Xu Yilun wrote:
[..]Quoted 3 lines
And I've no idea why unlocked MSIX/PBA must be hidden? How about other non-TEE ranges, must be hidden or mustn't? Is there a possibility we enforce DSM to present all ranges, then the layout is clear to OS?
Quoted 6 lines
Just to close this question, this was discussed at the last device security call. Indeed the expectation is that Linux will assume that all ranges besides MSIX/PBA must be present in the report, and that offset is always aligned. If / when an implementation violates that expectation they can help write the Linux quirk for that case or otherwise fix their implementation.
I don't think you can quirk it.
Implementations can always follow this requirement by setting the offset to 0. If they cannot compute a proper aligned offset then this is what they must do.
Jason
Aneesh Kumar K.V
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
In reply to Jason GunthorpeQuoted 11 lines · Jason Gunthorpe
Quoted 3 lines · Xu Yilun
My understanding is, it is the obfuscated host start pfn of this range, if this range has offset to the BAR start, this field should also be offsetted.
The OS must get an idea of the bar layout out of the report, so there have to be restrictions on how it is formed otherwise it is unparsible. IMHO the PCI spec created this very general mechanism but the CPU CC specs need to constrain it to be usable by an OS.
ARM CCA spec mention these restrictions in section
A9.6.2 Realm validation of device memory mappings
-aneesh
Alexey Kardashevskiy
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
In reply to Xu YilunQuoted 14 lines · Xu Yilun
Quoted 14 lines
Quoted 13 lines
drivers/pci/tsm/core.c +12
-0+if (last_bar < bar) { +/* transition to a new bar */ + last_bar = bar; +/* +* The tsm_offset for the first range of the BAR +* corresponds to the BAR base. +*/ + reporting_bar_base = tsm_offset; + }else if (tsm_offset < last_reporting_end) { +pci_dbg (pdev,"Reporting ranges within BAR not in ascending order\n" ); +return NULL ; + }....
Quoted 6 lines
Quoted 3 lines
drivers/pci/tsm/core.c static ssize_t dsm_show(struct device *dev, struct device_attribute *attr, · line 740
Quoted 2 lines
drivers/pci/tsm/core.c +1
-0+ range_off = tsm_offset - reporting_bar_base;
range_offwill always be zero? Should we do
tsm_offsetcomes from Device Interface Report, MMIO RANGE, First 4k Page.
It is not the first 4K though if the actual first 4K (or whatever) of that BAR is MSIX which the device is mandated to skip in the report if MSIX is not "locked".
Quoted 6 lines
How do you interpret the exact meaning of this field?
My understanding is, it is the obfuscated host start pfn of this range, if this range has offset to the BAR start, this field should also be offsetted.
Quoted 14 lines
But if the first range in the BAR should be aligned to BAR, otherwise there is no way for guest to position the range in the BAR.
Quoted 1 line
So the logic here is:
reporting_bar_base: the first obfuscated pfn for the BAR, the BAR pfn
tsm_offset: the current obfucated pfn for the BAR.
tsm_offset-reporting_bar_base: the offset to the BAR.Quoted 7 lines
range_off = tsm_offset & (pci_resource_len(pdev, bar) - 1);So that we correctly handle if the interface report is reporting a range within a bar. The only requirement here is bar address should be aligned to its size and
mmio_reporting_offsetshould not add offsets in that range.
and btw this only works if the entity generating the MMIO reporting offset (==TSM) knows about BARs sizes, which is not the case for AMD - the FW has no access to the config space (so the HV needs to feed this to the FW? may be). Thanks,
Signature · 1 line
Alexey
Jason Gunthorpe
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 4 lines · Alexey Kardashevskiy
and btw this only works if the entity generating the MMIO reporting offset (==TSM) knows about BARs sizes, which is not the case for AMD
- the FW has no access to the config space (so the HV needs to feed this to the FW? may be). Thanks,
Then your platform just shouldn't use the mmio offset feature. Set it to 0 always.
Jason
Alexey Kardashevskiy
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 9 lines · Jason Gunthorpe
Quoted 4 lines · Alexey Kardashevskiy
and btw this only works if the entity generating the MMIO reporting offset (==TSM) knows about BARs sizes, which is not the case for AMD
- the FW has no access to the config space (so the HV needs to feed this to the FW? may be). Thanks,Then your platform just shouldn't use the mmio offset feature. Set it to 0 always.
pcie r7, Table 11-16 TDI Report Structure, MMIO_RANGE:
"Each MMIO Range of the TDI is reported with the MMIO reporting offset added."
My english struggles here - can the above be interpreted as "Each reported MMIO Range ..."?
as if it is each (except msix), then I know where msix is and can amend the report inside the VM if msix is not locked. Thanks,
Signature · 1 line
Alexey
Jason Gunthorpe
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 13 lines · Alexey Kardashevskiy
Quoted 9 lines · Jason Gunthorpe
Quoted 4 lines · Alexey Kardashevskiy
and btw this only works if the entity generating the MMIO reporting offset (==TSM) knows about BARs sizes, which is not the case for AMD
- the FW has no access to the config space (so the HV needs to feed this to the FW? may be). Thanks,Then your platform just shouldn't use the mmio offset feature. Set it to 0 always.
Quoted 8 lines
pcie r7, Table 11-16 TDI Report Structure,
MMIO_RANGE:"Each MMIO Range of the TDI is reported with the MMIO reporting offset added."
My english struggles here - can the above be interpreted as "Each reported MMIO Range ..."?
as if it is each (except msix), then I know where msix is and can amend the report inside the VM if msix is not locked. Thanks,
To do this you must be convert between the offset'd and phys_addr_t versions otherwise you have no idea where the translated ones fall within the BAR, so you can't figure out if msix is covered or not.
Jason
Alexey Kardashevskiy
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 18 lines · Jason Gunthorpe
Quoted 17 lines · Alexey Kardashevskiy
Quoted 9 lines · Jason Gunthorpe
Quoted 4 lines · Alexey Kardashevskiy
and btw this only works if the entity generating the MMIO reporting offset (==TSM) knows about BARs sizes, which is not the case for AMD
- the FW has no access to the config space (so the HV needs to feed this to the FW? may be). Thanks,Then your platform just shouldn't use the mmio offset feature. Set it to 0 always.
pcie r7, Table 11-16 TDI Report Structure,
MMIO_RANGE:"Each MMIO Range of the TDI is reported with the MMIO reporting offset added."
Quoted 8 lines
Quoted 4 lines
My english struggles here - can the above be interpreted as "Each reported MMIO Range ..."?
as if it is each (except msix), then I know where msix is and can amend the report inside the VM if msix is not locked. Thanks,
To do this you must be convert between the offset'd and
phys_addr_tversions otherwise you have no idea where the translated ones fall within the BAR, so you can't figure out if msix is covered or not.
I know if MSIX is covered because I know (from the PSP) if it is locked so it must be reported, with specific MSIX/PBA flags. If it is not locked, then skipped in the report but I still know where it is. For other ranges, if the device is not skipping them randomly, then, with a preserved order (as PCIe mandates), it can be reconstructed.
PCIe:
===
MMIO ranges assigned via BAR(s) must be reported in ascending order starting with the lowest numbered BAR such that the first range corresponds to the first BAR and so on. The range ID reports the BAR equivalent Indicator (BEI). Values 0-7 of the Range ID are reserved to indicate the BEI. The device must report the BAR equivalent Indicator (BEI) for ranges associated with a PCIe BAR.
When reporting the MMIO range for a TDI, the MMIO ranges must be reported in the logical order in which the TDI MMIO range is configured such that the first range reported corresponds to first range of pages in the TDI and so on.
===
What do I miss? Thanks,
Signature · 1 line
Alexey
Jason Gunthorpe
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 1 line · Alexey Kardashevskiy
What do I miss? Thanks,
You can't tell where things start so there is no way to relate the offsets to something the kernel can understand.
Jason
Alexey Kardashevskiy
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 6 lines · Jason Gunthorpe
Quoted 1 line · Alexey Kardashevskiy
What do I miss? Thanks,
You can't tell where things start so there is no way to relate the offsets to something the kernel can understand.
Reported ranges have BAR indexes and start addresses (with the reported MMIO offset added), and the first reported range starts at the first 4K of that BAR. And these ranges are in sorted order. Enough to calculate an offset of a 2nd/3rd/... range of that BAR, and this is what I am after.
Signature · 1 line
Alexey
Jason Gunthorpe
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 5 lines · Alexey Kardashevskiy
Quoted 2 lines · Jason Gunthorpe
On Mon, Mar 30, 2026 at 04:47:44PM +1100, Alexey Kardashevskiy wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
What do I miss? Thanks,
You can't tell where things start so there is no way to relate the offsets to something the kernel can understand.
Reported ranges have BAR indexes and start addresses (with the reported MMIO offset added), and the first reported range starts at the first 4K of that BAR.
I was told this is not the case, the first reported range can start anywhere in the BAR?
Jason
Alexey Kardashevskiy
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 11 lines · Jason Gunthorpe
Quoted 10 lines · Alexey Kardashevskiy
Quoted 6 lines · Jason Gunthorpe
Quoted 1 line · Alexey Kardashevskiy
What do I miss? Thanks,
You can't tell where things start so there is no way to relate the offsets to something the kernel can understand.
Quoted 6 lines
Quoted 3 lines
Reported ranges have BAR indexes and start addresses (with the reported MMIO offset added), and the first reported range starts at the first 4K of that BAR.
I was told this is not the case, the first reported range can start anywhere in the BAR?
This is what I am trying to clarify - if all ranges must be reported (as some think this is what the PCIe spec says), then no, not anywhere.
pcie r7, Table 11-16 TDI Report Structure, MMIO_RANGE:
"Each MMIO Range of the TDI is reported with the MMIO reporting offset added."
Signature · 1 line
Alexey
Jason Gunthorpe
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 21 lines · Alexey Kardashevskiy
Quoted 11 lines · Jason Gunthorpe
Quoted 10 lines · Alexey Kardashevskiy
Quoted 6 lines · Jason Gunthorpe
Quoted 1 line · Alexey Kardashevskiy
What do I miss? Thanks,
You can't tell where things start so there is no way to relate the offsets to something the kernel can understand.
Quoted 6 lines
Quoted 3 lines
Reported ranges have BAR indexes and start addresses (with the reported MMIO offset added), and the first reported range starts at the first 4K of that BAR.
I was told this is not the case, the first reported range can start anywhere in the BAR?
Quoted 7 lines
This is what I am trying to clarify - if all ranges must be reported (as some think this is what the PCIe spec says), then no, not anywhere.
pcie r7, Table 11-16 TDI Report Structure,
MMIO_RANGE:"Each MMIO Range of the TDI is reported with the MMIO reporting offset added."
I think the argument was something like it didn't have to report non-secure ranges? But I don't know, it was hashed out in some thread for ARM and then I know our folks looked at it and nobody pushed back to insist that every single byte of the BAR had to be covered by a reported range.
I wouldn't take the sentance you quoted as confirmation, you need a sentance that says every single byte of the BAR is covered by a single reported range.
Jason
Alexey Kardashevskiy
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 28 lines · Jason Gunthorpe
Quoted 27 lines · Alexey Kardashevskiy
Quoted 11 lines · Jason Gunthorpe
Quoted 10 lines · Alexey Kardashevskiy
Quoted 6 lines · Jason Gunthorpe
Quoted 1 line · Alexey Kardashevskiy
What do I miss? Thanks,
You can't tell where things start so there is no way to relate the offsets to something the kernel can understand.
Quoted 6 lines
Quoted 3 lines
Reported ranges have BAR indexes and start addresses (with the reported MMIO offset added), and the first reported range starts at the first 4K of that BAR.
I was told this is not the case, the first reported range can start anywhere in the BAR?
This is what I am trying to clarify - if all ranges muI thinkst be reported (as some think this is what the PCIe spec says), then no, not anywhere.
pcie r7, Table 11-16 TDI Report Structure,
MMIO_RANGE:
Quoted 7 lines
Quoted 1 line
"Each MMIO Range of the TDI is reported with the MMIO reporting offset added."
I think the argument was something like it didn't have to report non-secure ranges? But I don't know, it was hashed out in some thread for ARM and then I know our folks looked at it and nobody pushed back to insist that every single byte of the BAR had to be covered by a reported range.
That's (my ignorant guess) because of the ARM FW TSM guy which sees the BARs and can easily make sure that MMIO_OFFSET is such that BAR alignment is preserved (and there is a clause in PCIe about how such offset is "permitted" to be calculated) => does not make much difference on ARM but it does in my case :-/
Quoted 3 lines
I wouldn't take the sentance you quoted as confirmation, you need a sentance that says every single byte of the BAR is covered by a single reported range.
Why "by a single range"? Every byte of a BAR needs to be covered (which is what my quote suggests) and the spec allows multiple ranges but also requires strict ascending order of the ranges, 3 paragraphs of text about it. Thanks,
Quoted 2 lines
Jason
Signature · 1 line
Alexey
Jason Gunthorpe
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 14 lines · Alexey Kardashevskiy
Quoted 6 lines
Quoted 6 lines
This is what I am trying to clarify - if all ranges muI thinkst be reported (as some think this is what the PCIe spec says), then no, not anywhere.
pcie r7, Table 11-16 TDI Report Structure,
MMIO_RANGE:Quoted 7 lines
Quoted 1 line
"Each MMIO Range of the TDI is reported with the MMIO reporting offset added."
I think the argument was something like it didn't have to report non-secure ranges? But I don't know, it was hashed out in some thread for ARM and then I know our folks looked at it and nobody pushed back to insist that every single byte of the BAR had to be covered by a reported range.
Quoted 7 lines
That's (my ignorant guess) because of the ARM FW TSM guy which sees the BARs and can easily make sure that
MMIO_OFFSETis such that BAR alignment is preserved (and there is a clause in PCIe about how such offset is "permitted" to be calculated) => does not make much difference on ARM but it does in my case :-/Quoted 3 lines
I wouldn't take the sentance you quoted as confirmation, you need a sentance that says every single byte of the BAR is covered by a single reported range.
Why "by a single range"? Every byte of a BAR needs to be covered (which is what my quote suggests)
No, your quote doesn't suggest that at all, it just says if a range is present it has to be offset.
In fact the spec specifically says not to report ranges sometimes:
Bit 0 - MSI-X Table - if the range maps MSI-X table. This must be reported **only if locked** by the LOCK_INTERFACE_REQUEST.
So if the MSI-X table is not locked then what is reported? Seems not covered by a range at all is the consensus answer.
Thus you get this case where the non-reported MSI-X table could be at byte 0, not get a range and then there is no range covering byte 0 of the bar at all.
Quoted 3 lines
and the spec allows multiple ranges but also requires strict ascending order of the ranges, 3 paragraphs of text about it. Thanks,
single range per byte means there are not overlapping ranges.
This was the old thread with my suggestion.
https://lore.kernel.org/all/[email protected]/
If this is important to AMD they need to get an ECN with PCI-SIG to clarify. I think as of right now Linux can't assume the ranges start at bar physical offset 0.
Jason
Alexey Kardashevskiy
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 20 lines · Jason Gunthorpe
Quoted 19 lines · Alexey Kardashevskiy
Quoted 6 lines
Quoted 6 lines
This is what I am trying to clarify - if all ranges muI thinkst be reported (as some think this is what the PCIe spec says), then no, not anywhere.
pcie r7, Table 11-16 TDI Report Structure,
MMIO_RANGE:Quoted 7 lines
Quoted 1 line
"Each MMIO Range of the TDI is reported with the MMIO reporting offset added."
I think the argument was something like it didn't have to report non-secure ranges? But I don't know, it was hashed out in some thread for ARM and then I know our folks looked at it and nobody pushed back to insist that every single byte of the BAR had to be covered by a reported range.
That's (my ignorant guess) because of the ARM FW TSM guy which sees the BARs and can easily make sure that
MMIO_OFFSETis such that BAR alignment is preserved (and there is a clause in PCIe about how such offset is "permitted" to be calculated) => does not make much difference on ARM but it does in my case :-/Quoted 3 lines
I wouldn't take the sentance you quoted as confirmation, you need a sentance that says every single byte of the BAR is covered by a single reported range.
Quoted 5 lines
Quoted 2 lines
Why "by a single range"? Every byte of a BAR needs to be covered (which is what my quote suggests)
No, your quote doesn't suggest that at all, it just says if a range is present it has to be offset.
At all? My hw architect says it does.
PCIe says "Each MMIO Range of the TDI is reported with the MMIO reporting offset added." Not "Each reported MMIO Range of the TDI is reported with the MMIO reporting offset added."
Quoted 6 lines
In fact the spec specifically says not to report ranges sometimes:
Bit 0 - MSI-X Table - if the range maps MSI-X table. This must be reported **only if locked** by the
LOCK_INTERFACE_REQUEST.
Quoted 6 lines
So if the MSI-X table is not locked then what is reported? Seems not covered by a range at all is the consensus answer.
Thus you get this case where the non-reported MSI-X table could be at byte 0, not get a range and then there is no range covering byte 0 of the bar at all.
This is the only case when dropping a range in the report is allowed and even required. When this happens, the OS knows MSIX is not locked (part of the FW ABI) and the OS knows where MSIX BAR is and can easily amend the report.
Quoted 5 lines
Quoted 3 lines
and the spec allows multiple ranges but also requires strict ascending order of the ranges, 3 paragraphs of text about it. Thanks,
single range per byte means there are not overlapping ranges.
ah, misread, sorry.
Quoted 7 lines
This was the old thread with my suggestion.
https://lore.kernel.org/all/[email protected]/
If this is important to AMD they need to get an ECN with PCI-SIG to clarify. I think as of right now Linux can't assume the ranges start at bar physical offset 0.
uff may be...
Quoted 2 lines
Jason
Signature · 1 line
Alexey
Jason Gunthorpe
maintainerRe: [PATCH v2 09/19] PCI/TSM: Support creating encrypted MMIO descriptors via TDISP Report
Quoted 23 lines · Alexey Kardashevskiy
Quoted 20 lines · Jason Gunthorpe
Quoted 19 lines · Alexey Kardashevskiy
Quoted 6 lines
Quoted 6 lines
This is what I am trying to clarify - if all ranges muI thinkst be reported (as some think this is what the PCIe spec says), then no, not anywhere.
pcie r7, Table 11-16 TDI Report Structure,
MMIO_RANGE:Quoted 7 lines
Quoted 1 line
"Each MMIO Range of the TDI is reported with the MMIO reporting offset added."
I think the argument was something like it didn't have to report non-secure ranges? But I don't know, it was hashed out in some thread for ARM and then I know our folks looked at it and nobody pushed back to insist that every single byte of the BAR had to be covered by a reported range.
That's (my ignorant guess) because of the ARM FW TSM guy which sees the BARs and can easily make sure that
MMIO_OFFSETis such that BAR alignment is preserved (and there is a clause in PCIe about how such offset is "permitted" to be calculated) => does not make much difference on ARM but it does in my case :-/Quoted 3 lines
I wouldn't take the sentance you quoted as confirmation, you need a sentance that says every single byte of the BAR is covered by a single reported range.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Why "by a single range"? Every byte of a BAR needs to be covered (which is what my quote suggests)
No, your quote doesn't suggest that at all, it just says if a range is present it has to be offset.
At all? My hw architect says it does.
I don't see how you can possibly read that phrase that way. Go ask your PCI SIG rep.
Quoted 2 lines
PCIe says "Each MMIO Range of the TDI is reported with the MMIO reporting offset added."
"MMIO Range" does not refer to an entire bar, it refers to an entry in the rage table.
Quoted 9 lines
Quoted 9 lines
In fact the spec specifically says not to report ranges sometimes:
Bit 0 - MSI-X Table - if the range maps MSI-X table. This must be reported **only if locked** by the
LOCK_INTERFACE_REQUEST.So if the MSI-X table is not locked then what is reported? Seems not covered by a range at all is the consensus answer.
Quoted 6 lines
Quoted 3 lines
Thus you get this case where the non-reported MSI-X table could be at byte 0, not get a range and then there is no range covering byte 0 of the bar at all.
This is the only case when dropping a range in the report is allowed and even required.
Which firmly disproves the assertion about the first phrase.
Quoted 3 lines
When this happens, the OS knows MSIX is not
locked (part of the FW ABI) and the OS knows where MSIX BAR is and can easily amend the report.
Typically OS has no idea how big MSIX things actually are, there is no way to fix things if the hole is at the start of the BAR, and that's a legal design.
Jason
PATCH v2 10/19 x86, swiotlb: Teach swiotlb to skip "accepted" devices
In reply to Dan WilliamsThere are two mechanisms to force SWIOTLB operation, the kernel command line option and the internal SWIOTLB_FORCE flag. With the arrival of "accepted" devices, devices that have been enabled to DMA to private encrypted memory, the SWIOTLB_FORCE flag is an awkward fit. It may be the case that SWIOTLB operation wants to be forced regardless of the device acceptance state.
Introduce a new SWIOTLB_UNACCPTED flag that allows for both augmenting the result of is_swiotlb_force_bounce() dynamically and allowing for an "always SWIOTLB" override.
--- include/linux/swiotlb.h | 15 ++++++++++++--- arch/x86/kernel/pci-dma.c | 2 +- kernel/dma/swiotlb.c | 1 + 3 files changed, 14 insertions(+), 4 deletions(-)
include/linux/swiotlb.h +12 -3
@@ -17,6 +17,7 @@ struct scatterlist;#define SWIOTLB_VERBOSE (1 << 0)/* verbose initialization */ #define SWIOTLB_FORCE (1 << 1)/* force bounce buffering */ #define SWIOTLB_ANY (1 << 2)/* allow any memory for the buffer */ +#define SWIOTLB_UNACCEPTED (1 << 3)/* swiotlb for unaccepted devices */ /* * Maximum allowable number of contiguous slabs to map,
@@ -91,6 +92,7 @@ struct io_tlb_pool {* @nslabs: Total number of IO TLB slabs in all pools. * @debugfs: The dentry to debugfs. * @force_bounce: %true if swiotlb bouncing is forced +* @bounce_unaccepted: %true if unaccepted devices must bounce * @for_alloc: %true if the pool is used for memory allocation * @can_grow: %true if more pools can be allocated dynamically. * @phys_limit: Maximum allowed physical address.
@@ -109,8 +111,9 @@ struct io_tlb_mem {struct io_tlb_pool defpool;unsigned long nslabs;struct dentry *debugfs; -bool force_bounce; -bool for_alloc; + u8 force_bounce:1; + u8 bounce_unaccepted:1; + u8 for_alloc:1;#ifdef CONFIG_SWIOTLB_DYNAMIC bool can_grow;u64 phys_limit;
@@ -173,7 +176,13 @@ static inline bool is_swiotlb_force_bounce(struct device *dev) {struct io_tlb_mem *mem = dev->dma_io_tlb_mem; -return mem && mem->force_bounce; +if (!mem) +return false; +if (mem->force_bounce) +return true; +if (mem->bounce_unaccepted && !device_cc_accepted (dev)) +return true; +return false; }void swiotlb_init (bool addressing_limited,unsigned int flags);
arch/x86/kernel/pci-dma.c +1 -1
@@ -61,7 +61,7 @@ static void __init pci_swiotlb_detect(void)*/ if (cc_platform_has (CC_ATTR_GUEST_MEM_ENCRYPT )) { x86_swiotlb_enable = true; - x86_swiotlb_flags |=SWIOTLB_FORCE ; + x86_swiotlb_flags |=SWIOTLB_UNACCEPTED ; } }#else
kernel/dma/swiotlb.c +1 -0
@@ -365,6 +365,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, io_tlb_default_mem.force_bounce = swiotlb_force_bounce || (flags &SWIOTLB_FORCE ); + io_tlb_default_mem.bounce_unaccepted = flags &SWIOTLB_UNACCEPTED ;#ifdef CONFIG_SWIOTLB_DYNAMIC if (!remap)
Signature · 1 line
2.52.0
Aneesh Kumar K.V
maintainerRe: [PATCH v2 10/19] x86, swiotlb: Teach swiotlb to skip "accepted" devices
Quoted 91 lines · Dan Williams
There are two mechanisms to force SWIOTLB operation, the kernel command line option and the internal
SWIOTLB_FORCEflag. With the arrival of "accepted" devices, devices that have been enabled to DMA to private encrypted memory, theSWIOTLB_FORCEflag is an awkward fit. It may be the case that SWIOTLB operation wants to be forced regardless of the device acceptance state.Introduce a new
SWIOTLB_UNACCPTEDflag that allows for both augmenting the result ofis_swiotlb_force_bounce()dynamically and allowing for an "always SWIOTLB" override.Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc Dave Hansen <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Marek Szyprowski <[email protected]>Cc Robin Murphy <[email protected]>Signed-off-by Dan Williams <[email protected]>--- include/linux/swiotlb.h | 15 ++++++++++++--- arch/x86/kernel/pci-dma.c | 2 +- kernel/dma/swiotlb.c | 1 + 3 files changed, 14 insertions(+), 4 deletions(-)include/linux/swiotlb.h +12
-3@@ -17,6 +17,7 @@ struct scatterlist;#define SWIOTLB_VERBOSE (1 << 0)/* verbose initialization */ #define SWIOTLB_FORCE (1 << 1)/* force bounce buffering */ #define SWIOTLB_ANY (1 << 2)/* allow any memory for the buffer */ +#define SWIOTLB_UNACCEPTED (1 << 3)/* swiotlb for unaccepted devices */ /* * Maximum allowable number of contiguous slabs to map, @@ -91,6 +92,7 @@ struct io_tlb_pool {* @nslabs: Total number of IO TLB slabs in all pools. * @debugfs: The dentry to debugfs. * @force_bounce: %true if swiotlb bouncing is forced +* @bounce_unaccepted: %true if unaccepted devices must bounce * @for_alloc: %true if the pool is used for memory allocation * @can_grow: %true if more pools can be allocated dynamically. * @phys_limit: Maximum allowed physical address. @@ -109,8 +111,9 @@ struct io_tlb_mem {struct io_tlb_pool defpool;unsigned long nslabs;struct dentry *debugfs; -bool force_bounce; -bool for_alloc; + u8 force_bounce:1; + u8 bounce_unaccepted:1; + u8 for_alloc:1;#ifdef CONFIG_SWIOTLB_DYNAMIC bool can_grow;u64 phys_limit;@@ -173,7 +176,13 @@ static inline bool is_swiotlb_force_bounce(struct device *dev) {struct io_tlb_mem *mem = dev->dma_io_tlb_mem; -return mem && mem->force_bounce; +if (!mem) +return false; +if (mem->force_bounce) +return true; +if (mem->bounce_unaccepted && !device_cc_accepted (dev)) +return true; +return false; }void swiotlb_init (bool addressing_limited,unsigned int flags);arch/x86/kernel/pci-dma.c +1
-1@@ -61,7 +61,7 @@ static void __init pci_swiotlb_detect(void)*/ if (cc_platform_has (CC_ATTR_GUEST_MEM_ENCRYPT )) { x86_swiotlb_enable = true; - x86_swiotlb_flags |=SWIOTLB_FORCE ; + x86_swiotlb_flags |=SWIOTLB_UNACCEPTED ; } }#else kernel/dma/swiotlb.c +0
-0
kernel/dma/swiotlb.c void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, · line 369
Quoted 5 lines
kernel/dma/swiotlb.c +1
-0@@ -365,6 +365,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, io_tlb_default_mem.force_bounce = swiotlb_force_bounce || (flags &SWIOTLB_FORCE ); + io_tlb_default_mem.bounce_unaccepted = flags &SWIOTLB_UNACCEPTED ;
This should be.
@@ -373,7 +373,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags,
io_tlb_default_mem.force_bounce =
swiotlb_force_bounce || (flags & SWIOTLB_FORCE);
- io_tlb_default_mem.bounce_unaccepted = flags & SWIOTLB_UNACCEPTED;
+ io_tlb_default_mem.bounce_unaccepted = !!(flags & SWIOTLB_UNACCEPTED);
Quoted 5 lines
kernel/dma/swiotlb.c +0
-0#ifdef CONFIG_SWIOTLB_DYNAMIC if (!remap)Signature · 1 line
2.52.0
Re: [PATCH v2 10/19] x86, swiotlb: Teach swiotlb to skip "accepted" devices
Quoted 10 lines
kernel/dma/swiotlb.c void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, · line 369
Quoted 6 lines
kernel/dma/swiotlb.c +1
-0@@ -365,6 +365,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, io_tlb_default_mem.force_bounce = swiotlb_force_bounce || (flags &SWIOTLB_FORCE ); + io_tlb_default_mem.bounce_unaccepted = flags &SWIOTLB_UNACCEPTED ;This should be.
kernel/dma/swiotlb.c void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, · line 376
Quoted 5 lines
kernel/dma/swiotlb.c +1
-1@@ -373,7 +373,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, io_tlb_default_mem.force_bounce = swiotlb_force_bounce || (flags &SWIOTLB_FORCE ); - io_tlb_default_mem.bounce_unaccepted = flags &SWIOTLB_UNACCEPTED ; + io_tlb_default_mem.bounce_unaccepted = !!(flags &SWIOTLB_UNACCEPTED );
Ah yes, I just realized assigning to a 1-bit field would truncate the assigned value to its LSB...
Aneesh Kumar K.V
maintainerRe: [PATCH v2 10/19] x86, swiotlb: Teach swiotlb to skip "accepted" devices
In reply to Dan WilliamsQuoted 79 lines · Dan Williams
There are two mechanisms to force SWIOTLB operation, the kernel command line option and the internal
SWIOTLB_FORCEflag. With the arrival of "accepted" devices, devices that have been enabled to DMA to private encrypted memory, theSWIOTLB_FORCEflag is an awkward fit. It may be the case that SWIOTLB operation wants to be forced regardless of the device acceptance state.Introduce a new
SWIOTLB_UNACCPTEDflag that allows for both augmenting the result ofis_swiotlb_force_bounce()dynamically and allowing for an "always SWIOTLB" override.Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc Dave Hansen <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Marek Szyprowski <[email protected]>Cc Robin Murphy <[email protected]>Signed-off-by Dan Williams <[email protected]>--- include/linux/swiotlb.h | 15 ++++++++++++--- arch/x86/kernel/pci-dma.c | 2 +- kernel/dma/swiotlb.c | 1 + 3 files changed, 14 insertions(+), 4 deletions(-)include/linux/swiotlb.h +12
-3@@ -17,6 +17,7 @@ struct scatterlist;#define SWIOTLB_VERBOSE (1 << 0)/* verbose initialization */ #define SWIOTLB_FORCE (1 << 1)/* force bounce buffering */ #define SWIOTLB_ANY (1 << 2)/* allow any memory for the buffer */ +#define SWIOTLB_UNACCEPTED (1 << 3)/* swiotlb for unaccepted devices */ /* * Maximum allowable number of contiguous slabs to map, @@ -91,6 +92,7 @@ struct io_tlb_pool {* @nslabs: Total number of IO TLB slabs in all pools. * @debugfs: The dentry to debugfs. * @force_bounce: %true if swiotlb bouncing is forced +* @bounce_unaccepted: %true if unaccepted devices must bounce * @for_alloc: %true if the pool is used for memory allocation * @can_grow: %true if more pools can be allocated dynamically. * @phys_limit: Maximum allowed physical address. @@ -109,8 +111,9 @@ struct io_tlb_mem {struct io_tlb_pool defpool;unsigned long nslabs;struct dentry *debugfs; -bool force_bounce; -bool for_alloc; + u8 force_bounce:1; + u8 bounce_unaccepted:1; + u8 for_alloc:1;#ifdef CONFIG_SWIOTLB_DYNAMIC bool can_grow;u64 phys_limit;@@ -173,7 +176,13 @@ static inline bool is_swiotlb_force_bounce(struct device *dev) {struct io_tlb_mem *mem = dev->dma_io_tlb_mem; -return mem && mem->force_bounce; +if (!mem) +return false; +if (mem->force_bounce) +return true; +if (mem->bounce_unaccepted && !device_cc_accepted (dev)) +return true; +return false; }void swiotlb_init (bool addressing_limited,unsigned int flags);arch/x86/kernel/pci-dma.c +0
-0
arch/x86/kernel/pci-dma.c static void __init pci_swiotlb_detect(void) · line 67
Quoted 8 lines
arch/x86/kernel/pci-dma.c +1
-1@@ -61,7 +61,7 @@ static void __init pci_swiotlb_detect(void)*/ if (cc_platform_has (CC_ATTR_GUEST_MEM_ENCRYPT )) { x86_swiotlb_enable = true; - x86_swiotlb_flags |=SWIOTLB_FORCE ; + x86_swiotlb_flags |=SWIOTLB_UNACCEPTED ; } }#else
I guess we can also include arm64 change here
modified arch/arm64/mm/init.c
@@ -335,7 +335,7 @@ void __init arch_mm_preinit(void)
if (is_realm_world()) {
swiotlb = true;
- flags |= SWIOTLB_FORCE;
+ flags |= SWIOTLB_UNACCEPTED;
}
if (IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) && !swiotlb) {
-aneesh
PATCH v2 11/19 x86, dma: Allow accepted devices to map private memory
In reply to Dan WilliamsWith the arrival of "accepted" devices, devices that have been enabled to DMA to private encrypted memory, coherent DMA allocation no longer requires page conversion. Update force_dma_unencrypted() to skip accepted devices.
--- arch/x86/mm/mem_encrypt.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/mm/mem_encrypt.c +3 -2
@@ -20,10 +20,11 @@bool force_dma_unencrypted (struct device *dev) {/* -* For SEV, all DMA must be to unencrypted addresses. +* Require unencrypted DMA unless the device has been "accepted", +* enabled by a TSM driver to DMA to private encrypted memory. */ if (cc_platform_has (CC_ATTR_GUEST_MEM_ENCRYPT )) -return true; +return !device_cc_accepted (dev);/* * For SME, all DMA must be to unencrypted addresses if the
Signature · 1 line
2.52.0
Re: [PATCH v2 11/19] x86, dma: Allow accepted devices to map private memory
Quoted 4 lines · Dan Williams
With the arrival of "accepted" devices, devices that have been enabled to DMA to private encrypted memory, coherent DMA allocation no longer requires page conversion. Update
force_dma_unencrypted()to skip accepted devices.
Quoted 9 lines
Cc Dave Hansen <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc "H. Peter Anvin" <[email protected]>Signed-off-by Dan Williams <[email protected]>
Quoted 22 lines
--- arch/x86/mm/mem_encrypt.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)arch/x86/mm/mem_encrypt.c +3
-2@@ -20,10 +20,11 @@bool force_dma_unencrypted (struct device *dev) {/* -* For SEV, all DMA must be to unencrypted addresses. +* Require unencrypted DMA unless the device has been "accepted", +* enabled by a TSM driver to DMA to private encrypted memory. */ if (cc_platform_has (CC_ATTR_GUEST_MEM_ENCRYPT )) -return true; +return !device_cc_accepted (dev);/* * For SME, all DMA must be to unencrypted addresses if the
Signature · 1 line
Alexey
PATCH v2 12/19 x86, ioremap, resource: Support IORES_DESC_ENCRYPTED for encrypted PCI MMIO
In reply to Dan WilliamsPCIe Trusted Execution Environment Device Interface Security Protocol (TDISP) arranges for a PCI device to support encrypted MMIO. In support of that capability, ioremap() needs a mechanism to detect when a PCI device has been dynamically transitioned into this secure state and enforce encrypted MMIO mappings.
Teach ioremap() about a new IORES_DESC_ENCRYPTED type that supplements the existing PCI Memory Space (MMIO) BAR resources. The proposal is that a resource, "PCI MMIO Encrypted", with this description type is injected by the PCI/TSM core for each PCI device BAR that is to be protected.
Unlike the existing encryption determination which is "implied with a silent fallback to an unencrypted mapping", this indication is "explicit with an expectation that the request fails instead of fallback". IORES_MUST_ENCRYPT is added to manage this expectation.
Given that "PCI MMIO Encrypted" is an additional resource in the tree, the IORESOURCE_BUSY flag will only be set on a descendant/child of that resource. That means it cannot share the same walk as the check for "System RAM". Add walk_iomem_res_desc() to check if any IORES_DESC_ENCRYPTED intersects the ioremap() range and set IORES_MUST_ENCRYPT accordingly. When IORES_MUST_ENCRYPT is set, the entire ioremap() range must be covered by IORES_DESC_ENCRYPTED.
--- include/linux/ioport.h | 1 + arch/x86/mm/ioremap.c | 49 +++++++++++++++++++++++++++++++----------- 2 files changed, 37 insertions(+), 13 deletions(-)
include/linux/ioport.h +1 -0
@@ -152,6 +152,7 @@ enum {enum {IORES_MAP_SYSTEM_RAM =BIT (0 ),IORES_MAP_ENCRYPTED =BIT (1 ), +IORES_MUST_ENCRYPT =BIT (2 ),/* disable transparent fallback */ };/* helpers to define resources */
arch/x86/mm/ioremap.c +36 -13
91 lines not on this page · raw message
Signature · 1 line
2.52.0
Borislav Petkov
maintainerRe: [PATCH v2 12/19] x86, ioremap, resource: Support IORES_DESC_ENCRYPTED for encrypted PCI MMIO
Quoted 11 lines · Dan Williams
PCIe Trusted Execution Environment Device Interface Security Protocol (TDISP) arranges for a PCI device to support encrypted MMIO. In support of that capability,
ioremap()needs a mechanism to detect when a PCI device has been dynamically transitioned into this secure state and enforce encrypted MMIO mappings.Teach
ioremap()about a newIORES_DESC_ENCRYPTEDtype that supplements the existing PCI Memory Space (MMIO) BAR resources. The proposal is that a resource, "PCI MMIO Encrypted", with this description type is injected by the PCI/TSM core for each PCI device BAR that is to be protected.
Quoted 4 lines
Unlike the existing encryption determination which is "implied with a silent fallback to an unencrypted mapping", this indication is "explicit with an expectation that the request fails instead of fallback".
IORES_MUST_ENCRYPTis added to manage this expectation.
"Add IORES_MUST_ENCRYPT to manage... "
Pls read section "2) Describe your changes" in Documentation/process/submitting-patches.rst for more details. Especially the part about ordering the code to do stuff.
Quoted 47 lines
Given that "PCI MMIO Encrypted" is an additional resource in the tree, the
IORESOURCE_BUSYflag will only be set on a descendant/child of that resource. That means it cannot share the same walk as the check for "System RAM". Addwalk_iomem_res_desc()to check if anyIORES_DESC_ENCRYPTEDintersects theioremap()range and setIORES_MUST_ENCRYPTaccordingly. WhenIORES_MUST_ENCRYPTis set, the entireioremap()range must be covered byIORES_DESC_ENCRYPTED.Cc Dave Hansen <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc "H. Peter Anvin" <[email protected]>Signed-off-by Dan Williams <[email protected]>--- include/linux/ioport.h | 1 + arch/x86/mm/ioremap.c | 49 +++++++++++++++++++++++++++++++----------- 2 files changed, 37 insertions(+), 13 deletions(-)include/linux/ioport.h +1
-0@@ -152,6 +152,7 @@ enum {enum {IORES_MAP_SYSTEM_RAM =BIT (0 ),IORES_MAP_ENCRYPTED =BIT (1 ), +IORES_MUST_ENCRYPT =BIT (2 ),/* disable transparent fallback */ };/* helpers to define resources */ arch/x86/mm/ioremap.c +1
-0@@ -36,6 +36,7 @@*/ struct ioremap_desc {unsigned int flags; +u64 encrypt_size; };/*
arch/x86/mm/ioremap.c static unsigned int __ioremap_check_ram(struct resource *res) · line 92
Quoted 6 lines
arch/x86/mm/ioremap.c +1
-2@@ -88,23 +89,35 @@ static unsigned int __ioremap_check_ram(struct resource *res) }/* -* In a SEV guest, NONE and RESERVED should not be mapped encrypted because -* there the whole memory is already encrypted. +* In a encrypted guest, NONE and RESERVED should not be mapped encrypted
"In an encrypted... "
Quoted 23 lines
arch/x86/mm/ioremap.c +13
-2+* because there the whole memory is already encrypted. +* +* For the encrypted case the entire range must agree with being mapped +* encrypted. */ -static unsigned int __ioremap_check_encrypted (struct resource *res) +static unsigned int __ioremap_check_encrypted (struct ioremap_desc *desc, +struct resource *res) { +u32 flags =0 ; + +if (res->desc ==IORES_DESC_ENCRYPTED ) + flags |=IORES_MUST_ENCRYPT ; +if (!cc_platform_has (CC_ATTR_GUEST_MEM_ENCRYPT )) -return 0 ; +return flags;switch (res->desc) {case IORES_DESC_NONE :case IORES_DESC_RESERVED :break ; +case IORES_DESC_ENCRYPTED :
arch/x86/mm/ioremap.c static unsigned int __ioremap_check_ram(struct resource *res) · line 121
Quoted 10 lines
arch/x86/mm/ioremap.c +4
-2+ desc->encrypt_size +=resource_size (res); + fallthrough;default : -return IORES_MAP_ENCRYPTED ; + flags |=IORES_MAP_ENCRYPTED ; } -return 0 ; +return flags; }
This looks too convoluted. I'm sure we can state the rules for encrypted ranges first and then can simplify it into something more understandable.
Quoted 8 lines
arch/x86/mm/ioremap.c +0
-2/* @@ -134,14 +147,10 @@ static int __ioremap_collect_map_flags(struct resource *res, void *arg) {struct ioremap_desc *desc = arg; -if (!(desc->flags &IORES_MAP_SYSTEM_RAM )) - desc->flags |=__ioremap_check_ram (res);
arch/x86/mm/ioremap.c static int __ioremap_collect_map_flags(struct resource *res, void *arg) · line 154
Quoted 10 lines
arch/x86/mm/ioremap.c +3
-5- -if (!(desc->flags &IORES_MAP_ENCRYPTED )) - desc->flags |=__ioremap_check_encrypted (res); + desc->flags |=__ioremap_check_ram (res); + desc->flags |=__ioremap_check_encrypted (desc, res); -return ((desc->flags & (IORES_MAP_SYSTEM_RAM |IORES_MAP_ENCRYPTED )) == - (IORES_MAP_SYSTEM_RAM |IORES_MAP_ENCRYPTED )); +return 0 ; }
This change I don't grok. Perhaps it needs to be a separate patch.
arch/x86/mm/ioremap.c static void __ioremap_check_mem(resource_size_t addr, unsigned long size, · line 173
Quoted 5 lines
arch/x86/mm/ioremap.c +0
-0/* @@ -162,6 +171,13 @@ static void __ioremap_check_mem(resource_size_t addr, unsigned long size,memset (desc,0 ,sizeof (struct ioremap_desc ));walk_mem_res (start, end, desc, __ioremap_collect_map_flags);
<---- newline here.
arch/x86/mm/ioremap.c static void __ioremap_check_mem(resource_size_t addr, unsigned long size, · line 180
Quoted 7 lines
arch/x86/mm/ioremap.c +7
-0+/* +* Encrypted MMIO may parent a driver's requested region, so it needs a +* separate search +*/ + desc->encrypt_size =0 ; +walk_iomem_res_desc (IORES_DESC_ENCRYPTED ,IORESOURCE_MEM , start, end, + desc, __ioremap_collect_map_flags);
Why is this here and executed unconditionally regardless of coco guest or not?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v2 13/19 samples/devsec: Introduce a PCI device-security bus + endpoint sample
In reply to Dan WilliamsEstablish just enough emulated PCI infrastructure to register a sample TSM (platform security manager) driver and have it discover an IDE + TEE (link encryption + device-interface security protocol (TDISP)) capable device.
Use the existing CONFIG_PCI_BRIDGE_EMUL to emulate an IDE capable root port, and open code the emulation of an endpoint device via simulated configuration cycle responses.
The devsec_link_tsm driver responds to the PCI core TSM operations as if it successfully exercised the given interface security protocol messages.
The devsec_bus and devsec_link_tsm drivers can be loaded in either order to reflect cases like SEV-TIO where the TSM is PCI-device firmware, and cases like TDX Connect where the TSM is a software agent running on the host CPU.
Follow-on patches exercise the IDE establishment core and the TDISP operations that guests execute. For now, just successfully complete setup and teardown of the DSM (device security manager) context as a building block for management of TDI (trusted device interface) instances.
Note that 2 host bridges are emulated to regression test a bug report
[1] against the initial TSM based IDE establishment.
Trimmed kernel logs from loading the devsec sample modules and connecting a device.
# modprobe devsec_bus
faux_driver devsec_bus: PCI host bridge to bus 10000:00
pci_bus 10000:00: root bus resource [bus 00-01]
pci_bus 10000:00: root bus resource [mem 0xf0000000-0xf03fffff]
pci 10000:00:00.0: [8086:ffff] type 01 class 0x060400 PCIe Root Port
pci 10000:00:00.0: PCI bridge to [bus 01]
pci 10000:00:00.0: bridge window [mem 0xf0000000-0xf03fffff]
pci 10000:01:00.0: [8086:ffff] type 00 class 0x120000 PCIe Endpoint
pci 10000:01:00.1: [8086:ffff] type 00 class 0x120000 PCIe Endpoint
pci 10000:01:00.0: BAR 0 [mem 0xf0000000-0xf01fffff]: assigned
pci 10000:01:00.1: BAR 0 [mem 0xf0200000-0xf03fffff]: assigned
pcieport 10000:00:00.0: enabling device (0000 -> 0002)
faux_driver devsec_bus: PCI host bridge to bus 10001:02
pci_bus 10001:02: root bus resource [bus 02-03]
pci_bus 10001:02: root bus resource [mem 0xf0400000-0xf07fffff]
pci 10001:02:00.0: [8086:ffff] type 01 class 0x060400 PCIe Root Port
pci 10001:02:00.0: PCI bridge to [bus 03]
pci 10001:02:00.0: bridge window [mem 0xf0400000-0xf07fffff]
pci 10001:03:00.0: [8086:ffff] type 00 class 0x120000 PCIe Endpoint
pci 10001:03:00.1: [8086:ffff] type 00 class 0x120000 PCIe Endpoint
pci 10001:03:00.0: BAR 0 [mem 0xf0400000-0xf05fffff]: assigned
pci 10001:03:00.1: BAR 0 [mem 0xf0600000-0xf07fffff]: assigned
pcieport 10001:02:00.0: enabling device (0000 -> 0002) # modprobe devsec_link_tsm
link_sysfs_enable: pci 10000:01:00.0: PCI/TSM: Platform TEE Security Manager detected (IDE TEE)
link_sysfs_enable: pci 10001:03:00.0: PCI/TSM: Platform TEE Security Manager detected (IDE TEE) # echo tsm0 > /sys/bus/pci/devices/10000:01:00.0/tsm/connect
devsec_tsm_pf0_probe: pci 10000:01:00.0: devsec: TSM enabled
link_sysfs_enable: pci 10000:01:00.1: PCI/TSM: Device Security Manager detected (TEE)--- samples/Kconfig | 19 + samples/Makefile | 1 + samples/devsec/Makefile | 10 + samples/devsec/devsec.h | 43 +++ samples/devsec/bus.c | 780 ++++++++++++++++++++++++++++++++++++++ samples/devsec/common.c | 28 ++ samples/devsec/link_tsm.c | 192 ++++++++++ MAINTAINERS | 1 + 8 files changed, 1074 insertions(+) create mode 100644 samples/devsec/Makefile create mode 100644 samples/devsec/devsec.h create mode 100644 samples/devsec/bus.c create mode 100644 samples/devsec/common.c create mode 100644 samples/devsec/link_tsm.c
samples/Kconfig +19 -0
@@ -324,6 +324,25 @@ source "samples/rust/Kconfig" source "samples/damon/Kconfig" +config SAMPLE_DEVSEC + tristate "Build a sample TEE Security Manager with an emulated PCI endpoint" + depends on m + depends on PCI + depends on VIRT_DRIVERS + depends on PCI_DOMAINS + select PCI_BRIDGE_EMUL + select PCI_TSM + select TSM + help + Build a sample platform TEE Security Manager (TSM) driver with a + corresponding emulated PCIe topology. The resulting sample modules, + devsec_bus and devsec_tsm, exercise device-security enumeration, PCI + subsystem use ABIs, device security flows. For example, exercise IDE + (link encryption) establishment and TDISP state transitions via a + Device Security Manager (DSM). + + If unsure, say N + endif # SAMPLES config HAVE_SAMPLE_FTRACE_DIRECT
samples/Makefile +1 -0
@@ -45,3 +45,4 @@ obj-$(CONFIG_SAMPLE_DAMON_PRCL) += damon/ obj-$(CONFIG_SAMPLE_DAMON_MTIER) += damon/ obj-$(CONFIG_SAMPLE_HUNG_TASK) += hung_task/ obj-$(CONFIG_SAMPLE_TSM_MR) += tsm-mr/ +obj-y += devsec/
samples/devsec/Makefile new file +10 -0
@@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_SAMPLE_DEVSEC) += devsec_common.o +devsec_common-y := common.o + +obj-$(CONFIG_SAMPLE_DEVSEC) += devsec_bus.o +devsec_bus-y := bus.o + +obj-$(CONFIG_SAMPLE_DEVSEC) += devsec_link_tsm.o +devsec_link_tsm-y := link_tsm.o
samples/devsec/devsec.h new file +43 -0
43 lines not on this page · raw message
samples/devsec/bus.c new file +780 -0
780 lines not on this page · raw message
samples/devsec/common.c new file +28 -0
@@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (C) 2024 - 2026 Intel Corporation */ + +#include <linux/pci.h> +#include <linux/export.h> + +#include "devsec.h" + +/* +* devsec_bus and devsec_tsm need a common location for this data to +* avoid depending on each other. Enables load order testing +*/ +struct devsec_sysdata *devsec_sysdata[NR_DEVSEC_HOST_BRIDGES ]; +EXPORT_SYMBOL_FOR_MODULES (devsec_sysdata,"devsec*" ); + +static int __init common_init (void ) +{ +return 0 ; +} +module_init (common_init); + +static void __exit common_exit(void) +{ +} +module_exit (common_exit); + +MODULE_LICENSE ("GPL" ); +MODULE_DESCRIPTION ("Device Security Sample Infrastructure: Shared data" );
samples/devsec/link_tsm.c new file +192 -0
192 lines not on this page · raw message
MAINTAINERS +1 -0
@@ -26539,6 +26539,7 @@ F: drivers/pci/tsm/ F: drivers/virt/coco/guest/ F: include/linux/*tsm*.h F: include/uapi/linux/pci-tsm-netlink.h +F: samples/devsec/ F: samples/tsm-mr/ TRUSTED SERVICES TEE DRIVER
Signature · 1 line
2.52.0
PATCH v2 14/19 samples/devsec: Add sample IDE establishment
In reply to Dan WilliamsExercise common setup and teardown flows for a sample platform TSM driver that implements the TSM 'connect' and 'disconnect' flows.
This is both a template for platform specific implementations and a simple integration test for the PCI core infrastructure + ABI.
--- samples/devsec/bus.c | 4 +++ samples/devsec/link_tsm.c | 70 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 73 insertions(+), 1 deletion(-)
samples/devsec/bus.c +4 -0
@@ -6,6 +6,7 @@#include <linux/device/faux.h> #include <linux/module.h> #include <linux/pci.h> +#include <linux/pci-ide.h> #include <linux/range.h> #include <uapi/linux/pci_regs.h>
@@ -16,6 +17,7 @@#define NR_DEVICES_PER_BUS 1#define NR_FUNCTIONS 2#define NR_DEVICES (NR_DEVICES_PER_BUS * NR_FUNCTIONS * NR_DEVSEC_BUSES) +#define NR_HOST_BRIDGE_STREAMS 4#define NR_PORT_STREAMS 1#define NR_PORT_LINK_STREAMS 4#define NR_ADDR_ASSOC 1
@@ -728,6 +730,7 @@ static int devsec_add_host_bridge(struct faux_device *fdev, int hb_idx) hb->dev.parent = dev; hb->sysdata = sd; hb->ops = &devsec_ops; +pci_ide_set_nr_streams (hb,NR_HOST_BRIDGE_STREAMS ); rc =pci_scan_root_bus_bridge (hb);if (rc)
@@ -776,5 +779,6 @@ static void __exit devsec_bus_exit(void) }module_exit (devsec_bus_exit); +MODULE_IMPORT_NS ("PCI_IDE" );MODULE_LICENSE ("GPL" );MODULE_DESCRIPTION ("Device Security Sample Infrastructure: TDISP Device Emulation" );
samples/devsec/link_tsm.c +69 -1
92 lines not on this page · raw message
Signature · 1 line
2.52.0
PATCH v2 15/19 samples/devsec: Add sample TSM bind and guest_request flows
In reply to Dan WilliamsProvide a method to test the basic object lifetime mechanics of 'struct pci_tdi', and passthrough sysfs message to simulate pci_tsm_guest_req(). Arrange for pci_tsm_bind() and pci_tsm_guest_req() to be invoked via devsec_link_tsm faux-device sysfs attributes.
--- Documentation/ABI/testing/sysfs-faux-devsec | 15 ++ samples/devsec/link_tsm.c | 157 +++++++++++++++++++- 2 files changed, 170 insertions(+), 2 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-faux-devsec
Documentation/ABI/testing/sysfs-faux-devsec new file +15 -0
@@ -0,0 +1,15 @@ +What: /sys/bus/faux/devices/{devsec_link_tsm,devsec_tsm} +Contact: [email protected] +Description: + (DIR) The devsec_link_tsm and devsec_tsm faux devices test the + kernel's ABIs and flows that support PCIe Trusted Device + Interface Security Protocol (TDISP). The devsec_link_tsm device + simulates a "host" TSM that establishes an SPDM session and link + security (PCIe IDE). The devsec_tsm device simulates a "guest" + TSM that implements the lock+accept flows. + +What: +/sys/bus/faux/devices/devsec_link_tsm/{bind,unbind} +Contact: [email protected] +Description: +
samples/devsec/link_tsm.c +155 -2
200 lines not on this page · raw message
Signature · 1 line
2.52.0
PATCH v2 16/19 samples/devsec: Introduce a "Device Security TSM" sample driver
In reply to Dan WilliamsThere are 2 sides to a TEE Security Manager (TSM), the 'link' TSM, and the 'devsec' TSM. The 'link' TSM, outside the TEE, establishes physical link confidentiality and integerity, and a secure session for transporting commands the manage the security state of devices. The 'devsec' TSM, within the TEE, issues requests for confidential devices to lock their configuration and transition to secure operation.
Implement a sample implementation of a 'devsec' TSM. This leverages the PCI core's ability to register multiple TSMs at a time to load a sample devsec_tsm module alongside the existing devsec_link_tsm module. When both are loaded the TSM personality is selected by choosing to 'connect' vs 'lock' the device.
Drivers like tdx_guest, sev_guest, or arm-cca-guest are examples of "Device Security TSM" drivers.
A devsec_pci driver is included to test the device_cc_probe() helper for drivers that need to coordinate some configuration before 'lock' and 'accept'.
--- samples/devsec/Makefile | 6 ++ samples/devsec/pci.c | 39 +++++++++++++ samples/devsec/tsm.c | 124 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 169 insertions(+) create mode 100644 samples/devsec/pci.c create mode 100644 samples/devsec/tsm.c
samples/devsec/Makefile +6 -0
@@ -8,3 +8,9 @@ devsec_bus-y := bus.o obj-$(CONFIG_SAMPLE_DEVSEC) += devsec_link_tsm.o devsec_link_tsm-y := link_tsm.o + +obj-$(CONFIG_SAMPLE_DEVSEC) += devsec_tsm.o +devsec_tsm-y := tsm.o + +obj-$(CONFIG_SAMPLE_DEVSEC) += devsec_pci.o +devsec_pci-y := pci.o
samples/devsec/pci.c new file +39 -0
@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (C) 2024 - 2026 Intel Corporation */ +#include <linux/device.h> +#include <linux/module.h> +#include <linux/pci.h> + +static int devsec_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *id) +{ +void __iomem *base; +int rc; + + rc =pcim_enable_device (pdev); +if (rc) +return dev_err_probe (&pdev->dev, rc,"enable failed\n" ); + + base =pcim_iomap_region (pdev,0 ,KBUILD_MODNAME ); +if (IS_ERR (base)) +return dev_err_probe (&pdev->dev,PTR_ERR (base), +"iomap failed\n" ); + +dev_dbg (&pdev->dev,"attach\n" ); +return 0 ; +} + +static const struct pci_device_id devsec_pci_ids[] = { + {PCI_DEVICE (0x8086 ,0xffff ), .override_only =1 , }, + { } +}; + +static struct pci_driver devsec_pci_driver = { + .name ="devsec_pci" , + .probe = devsec_pci_probe, + .id_table = devsec_pci_ids, +}; + +module_pci_driver (devsec_pci_driver); +MODULE_LICENSE ("GPL" ); +MODULE_DESCRIPTION ("Device Security Sample Infrastructure: Secure PCI Driver" );
samples/devsec/tsm.c new file +124 -0
124 lines not on this page · raw message
Signature · 1 line
2.52.0
Re: [PATCH v2 16/19] samples/devsec: Introduce a "Device Security TSM" sample driver
Quoted 130 lines · Dan Williams
There are 2 sides to a TEE Security Manager (TSM), the 'link' TSM, and the 'devsec' TSM. The 'link' TSM, outside the TEE, establishes physical link confidentiality and integerity, and a secure session for transporting commands the manage the security state of devices. The 'devsec' TSM, within the TEE, issues requests for confidential devices to lock their configuration and transition to secure operation.
Implement a sample implementation of a 'devsec' TSM. This leverages the PCI core's ability to register multiple TSMs at a time to load a sample
devsec_tsmmodule alongside the existingdevsec_link_tsmmodule. When both are loaded the TSM personality is selected by choosing to 'connect' vs 'lock' the device.Drivers like
tdx_guest,sev_guest, or arm-cca-guest are examples of "Device Security TSM" drivers.A
devsec_pcidriver is included to test thedevice_cc_probe()helper for drivers that need to coordinate some configuration before 'lock' and 'accept'.Signed-off-by Dan Williams <[email protected]>--- samples/devsec/Makefile | 6 ++ samples/devsec/pci.c | 39 +++++++++++++ samples/devsec/tsm.c | 124 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 169 insertions(+) create mode 100644 samples/devsec/pci.c create mode 100644 samples/devsec/tsm.csamples/devsec/Makefile +6
-0@@ -8,3 +8,9 @@ devsec_bus-y := bus.o obj-$(CONFIG_SAMPLE_DEVSEC) += devsec_link_tsm.o devsec_link_tsm-y := link_tsm.o + +obj-$(CONFIG_SAMPLE_DEVSEC) += devsec_tsm.o +devsec_tsm-y := tsm.o + +obj-$(CONFIG_SAMPLE_DEVSEC) += devsec_pci.o +devsec_pci-y := pci.osamples/devsec/pci.c new file +39
-0@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (C) 2024 - 2026 Intel Corporation */ +#include <linux/device.h> +#include <linux/module.h> +#include <linux/pci.h> + +static int devsec_pci_probe (struct pci_dev *pdev, +const struct pci_device_id *id) +{ +void __iomem *base; +int rc; + + rc =pcim_enable_device (pdev); +if (rc) +return dev_err_probe (&pdev->dev, rc,"enable failed\n" ); + + base =pcim_iomap_region (pdev,0 ,KBUILD_MODNAME ); +if (IS_ERR (base)) +return dev_err_probe (&pdev->dev,PTR_ERR (base), +"iomap failed\n" ); + +dev_dbg (&pdev->dev,"attach\n" ); +return 0 ; +} + +static const struct pci_device_id devsec_pci_ids[] = { + {PCI_DEVICE (0x8086 ,0xffff ), .override_only =1 , }, + { } +}; + +static struct pci_driver devsec_pci_driver = { + .name ="devsec_pci" , + .probe = devsec_pci_probe, + .id_table = devsec_pci_ids, +}; + +module_pci_driver (devsec_pci_driver); +MODULE_LICENSE ("GPL" ); +MODULE_DESCRIPTION ("Device Security Sample Infrastructure: Secure PCI Driver" );samples/devsec/tsm.c new file +36
-0@@ -0,0 +1,124 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (C) 2024 - 2026 Intel Corporation */ + +#define dev_fmt (fmt) "devsec: " fmt +#include <linux/device/faux.h> +#include <linux/pci-tsm.h> +#include <linux/module.h> +#include <linux/pci.h> +#include <linux/tsm.h> +#include "devsec.h" + +struct devsec_dev_data { +struct pci_tsm_devsec pci; +}; + +static struct devsec_dev_data *to_devsec_data (struct pci_tsm *tsm) +{ + return container_of(tsm, struct devsec_dev_data, pci.base_tsm); +} + +static struct pci_tsm *devsec_tsm_lock (struct tsm_dev *tsm_dev,struct pci_dev *pdev) +{ +int rc; + + struct devsec_dev_data *devsec_data __free(kfree) = +kzalloc (sizeof (*devsec_data),GFP_KERNEL ); +if (!devsec_data) +return ERR_PTR (-ENOMEM ); + + rc =pci_tsm_devsec_constructor (pdev, &devsec_data->pci, tsm_dev); +if (rc) +return ERR_PTR (rc); + +return &no_free_ptr (devsec_data)->pci.base_tsm; +} +
samples/devsec/tsm.c line 46
Quoted 10 lines
samples/devsec/tsm.c new file +10
-0+static void devsec_tsm_unlock (struct pci_tsm *tsm) +{ +struct devsec_dev_data *devsec_data =to_devsec_data (tsm); +struct pci_tsm_devsec *devsec_tsm =to_pci_tsm_devsec (tsm); + +pci_tsm_mmio_teardown (devsec_tsm->mmio); +kfree (devsec_tsm->mmio); +kfree (devsec_data); +} +
Hi Dan,
While validating devsec mode transitions, I hit a reproducible crash in the sample devsec driver.
Reproducer:
1. lock with devsec tsm
2. unlock
Observed: NULL pointer dereference in the MMIO teardown path
Expected: unlock from LOCKED should return to UNLOCKED safely.
My understanding is that this is a sample driver implementation bug - missing NULL guard before MMIO teardown.
A follow-up question: do you prefer current design and each device security TSM driver is responsible for MMIO check, or should tsm/core adds a NULL guard to avoid potential crash?
Regards,
Yi Lai
Quoted 81 lines
samples/devsec/tsm.c new file +78
-0+static int devsec_tsm_accept (struct pci_dev *pdev) +{ +struct pci_tsm_devsec *devsec_tsm =to_pci_tsm_devsec (pdev->tsm); +int rc; + + struct pci_tsm_mmio *mmio __free(kfree) = +kzalloc (struct_size (mmio, mmio,PCI_NUM_RESOURCES ),GFP_KERNEL ); +if (!mmio) +return -ENOMEM ; + +/* +* Typically this range request would come from the TDISP Interface +* Report. For this sample, just request all BARs be marked encrypted +*/ +for (int i =0 ; i <PCI_NUM_RESOURCES ; i++) { +struct resource *res =pci_tsm_mmio_resource (mmio, mmio->nr); + +if (pci_resource_len (pdev, i) ==0 || + !(pci_resource_flags (pdev, i) &IORESOURCE_MEM )) +continue ; + res->start =pci_resource_start (pdev, i); + res->end =pci_resource_end (pdev, i); + mmio->nr++; + } + + rc =pci_tsm_mmio_setup (pdev, mmio); +if (rc) +return rc; + devsec_tsm->mmio =no_free_ptr (mmio); +return 0 ; +} + +static struct pci_tsm_ops devsec_pci_ops = { + .lock = devsec_tsm_lock, + .unlock = devsec_tsm_unlock, + .accept = devsec_tsm_accept, +}; + +static void devsec_tsm_remove (void *tsm_dev) +{ +tsm_unregister (tsm_dev); +} + +static int devsec_tsm_probe (struct faux_device *fdev) +{ +struct tsm_dev *tsm_dev; + + tsm_dev =tsm_register (&fdev->dev, &devsec_pci_ops); +if (IS_ERR (tsm_dev)) +return PTR_ERR (tsm_dev); + +return devm_add_action_or_reset (&fdev->dev, devsec_tsm_remove, + tsm_dev); +} + +static struct faux_device *devsec_tsm; + +static const struct faux_device_ops devsec_device_ops = { + .probe = devsec_tsm_probe, +}; + +static int __init devsec_tsm_init (void ) +{ + devsec_tsm =faux_device_create ("devsec_tsm" ,NULL , &devsec_device_ops); +if (!devsec_tsm) +return -ENOMEM ; +return 0 ; +} +module_init (devsec_tsm_init); + +static void __exit devsec_tsm_exit(void) +{ +faux_device_destroy (devsec_tsm); +} +module_exit (devsec_tsm_exit); + +MODULE_LICENSE ("GPL" ); +MODULE_DESCRIPTION ("Device Security Sample Infrastructure: Device Security TSM Driver" );Signature · 1 line
2.52.0
PATCH v2 17/19 tools/testing/devsec: Add a script to exercise samples/devsec/
In reply to Dan WilliamsRun the samples/devsec/ infrastructure through the PCIe TDISP connect, bind, lock, and accept flows. Include tests for module "autoprobe" policy.
--- tools/testing/devsec/devsec.sh | 234 +++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 2 files changed, 235 insertions(+) create mode 100755 tools/testing/devsec/devsec.sh
tools/testing/devsec/devsec.sh new file +234 -0
@@ -0,0 +1,234 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2025-2026 Intel Corporation + +# Checkout PCI/TSM sysfs and driver-core mechanics with the +# devsec_link_tsm and devsec_tsm sample modules from samples/devsec/. + +set -ex + +trap 'err $LINENO' ERR +err() { + echo $(basename $0): failed at line $1 + [ -n "$2" ] && "$2" + exit 1 +} + +ORDER="" + +setup_modules() { + if [[ $ORDER == "bus" ]]; then + modprobe devsec_bus + modprobe devsec_link_tsm + modprobe devsec_tsm + else + modprobe devsec_tsm + modprobe devsec_link_tsm + modprobe devsec_bus + fi +} + +teardown_modules() { + if [[ $ORDER == "bus" ]]; then + modprobe -r devsec_tsm + modprobe -r devsec_link_tsm + modprobe -r devsec_bus + else + modprobe -r devsec_bus + modprobe -r devsec_link_tsm + modprobe -r devsec_tsm + fi +} + +PCI_DEVS=( +"/sys/bus/pci/devices/10000:01:00.0" +"/sys/bus/pci/devices/10001:03:00.0" +) +FN_DEVS=( +"/sys/bus/pci/devices/10000:01:00.1" +"/sys/bus/pci/devices/10001:03:00.1" +) +tsm_devsec="" +tsm_link="" +devsec_pci="/sys/bus/pci/drivers/devsec_pci" + +tdisp_test() { + pci_dev=${PCI_DEVS[$1]} + fn_dev=${FN_DEVS[$1]} + host_bridge=$(dirname $(dirname $(readlink -f $pci_dev))) + + # with the device disconnected from the devsec TSM validate that + # the devsec_pci driver loads and honors the autoprobe policy + echo "devsec_pci" > $pci_dev/driver_override + modprobe devsec_pci "autoprobe=0" + + [[ -e $pci_dev/driver ]] && err "$LINENO" + echo $(basename $pci_dev) > $devsec_pci/bind + echo $(basename $pci_dev) > $devsec_pci/unbind + + # grab the device's resource from /proc/iomem + resource=$(cat /proc/iomem | grep -m1 $(basename $pci_dev) | awk -F ' :' '{print $1}' | tr -d ' ') + [[ -n $resource ]] || err "$LINENO" + + # lock and accept the device, validate that the resource is now + # marked encrypted + echo $(basename $tsm_devsec) > $pci_dev/tsm/lock + echo 1 > $pci_dev/tsm/accept + + cat /proc/iomem | grep "$resource" | grep -q -m1 "PCI MMIO Encrypted" || err "$LINENO" + + # validate that the driver now fails with -EINVAL when trying to + # bind + expect="echo: write error: Invalid argument" + echo $(basename $pci_dev) 2>&1 > $devsec_pci/bind | grep -q "$expect" || err "$LINENO" + + # unlock and validate that the encrypted mmio is removed + echo $(basename $tsm_devsec) > $pci_dev/tsm/unlock + cat /proc/iomem | grep "$resource" | grep -q "PCI MMIO Encrypted" && err "$LINENO" + + modprobe -r devsec_pci +} + +validate_disconnected() { + pci_dev=${PCI_DEVS[$1]} + fn_dev=${FN_DEVS[$1]} + host_bridge=$(dirname $(dirname $(readlink -f $pci_dev))) + + # validate that the dsm is not yet detected and that the sub-function + # is aware of any TSM capabilities + dsm=$(cat $pci_dev/tsm/dsm) || err "$LINENO from $2" + bound=$(cat $pci_dev/tsm/bound) || err "$LINENO from $2" + [[ -z $dsm ]] || err "$LINENO from $2" + [[ -z $bound ]] || err "$LINENO from $2" + [[ ! -e $fn_dev/tsm/dsm ]] || err "$LINENO from $2" + [[ ! -e $fn_dev/tsm/bound ]] || err "$LINENO from $2" + [[ ! -e $fn_dev/tsm/connect ]] || err "$LINENO from $2" + [[ ! -e $fn_dev/tsm/disconnect ]] || err "$LINENO from $2" +} + +# check that all devices can be connected simultaneously +ide_multi_test() { + for pci_dev in ${PCI_DEVS[@]}; do + echo $(basename $tsm_link) > $pci_dev/tsm/connect + done + + #check stream links show up and point back to the pci_dev + for pci_dev in ${PCI_DEVS[@]}; do + host_bridge=$(dirname $(dirname $(readlink -f $pci_dev))) + hb=$(basename $host_bridge) + [[ -e $host_bridge/stream0.0.0 ]] || err "$LINENO" + [[ -e $tsm_link/$hb/stream0.0.0 ]] || err "$LINENO" + [[ $(readlink -f "$tsm_link/$hb/stream0.0.0") == $(readlink -f $pci_dev) ]] || err "$LINENO" + done + + for pci_dev in ${PCI_DEVS[@]}; do + echo $(basename $tsm_link) > $pci_dev/tsm/disconnect + done +} + +ide_test() { + pci_dev=${PCI_DEVS[$1]} + fn_dev=${FN_DEVS[$1]} + host_bridge=$(dirname $(dirname $(readlink -f $pci_dev))) + + # validate that all of the secure streams are idle by default + hb=$(basename $host_bridge) + nr=$(cat $host_bridge/available_secure_streams) + [[ $nr == 4 ]] || err "$LINENO" + + validate_disconnected $1 $LINENO + + # connect a stream and validate that the stream link shows up at + # the host bridge and the TSM + echo $(basename $tsm_link) > $pci_dev/tsm/connect + nr=$(cat $host_bridge/available_secure_streams) + [[ $nr == 3 ]] || err "$LINENO" + + [[ $(cat $pci_dev/tsm/connect) == $(basename $tsm_link) ]] || err "$LINENO" + [[ -e $host_bridge/stream0.0.0 ]] || err "$LINENO" + [[ -e $tsm_link/$hb/stream0.0.0 ]] || err "$LINENO" + + # with the DSM connected (PF0), validate both it and its + # sub-function (PF1) populate tsm/dsm with the PF0 device. + dsm=$(cat $pci_dev/tsm/dsm) + [[ $dsm == $(basename $pci_dev) ]] || err "$LINENO" + dsm=$(cat $fn_dev/tsm/dsm) + [[ $dsm == $(basename $pci_dev) ]] || err "$LINENO" + + # bind both functions and validate that they display bound to + # the TSM device + echo $(basename $pci_dev) > $tsm_link/device/tsm_bind + bound=$(cat $pci_dev/tsm/bound) + [[ $bound == $(basename $tsm_link) ]] || err "$LINENO" + echo $(basename $fn_dev) > $tsm_link/device/tsm_bind + bound=$(cat $fn_dev/tsm/bound) + [[ $bound == $(basename $tsm_link) ]] || err "$LINENO" + + # test manual unbind + echo $(basename $pci_dev) > $tsm_link/device/tsm_unbind + bound=$(cat $pci_dev/tsm/bound) + [[ -z $bound ]] || err "$LINENO" + echo $(basename $fn_dev) > $tsm_link/device/tsm_unbind + bound=$(cat $fn_dev/tsm/bound) + [[ -z $bound ]] || err "$LINENO" + + # rebind to test automatic unbind at disconnect + echo $(basename $pci_dev) > $tsm_link/device/tsm_bind + echo $(basename $fn_dev) > $tsm_link/device/tsm_bind + + # check that the links disappear at disconnect and the stream + # pool is refilled + echo $(basename $tsm_link) > $pci_dev/tsm/disconnect + nr=$(cat $host_bridge/available_secure_streams) + [[ $nr == 4 ]] || err "$LINENO" + + validate_disconnected $1 $LINENO + + [[ $(cat $pci_dev/tsm/connect) == "" ]] || err "$LINENO" + [[ ! -e $host_bridge/stream0.0.0 ]] || err "$LINENO" + [[ ! -e $tsm_link/$hb/stream0.0.0 ]] || err "$LINENO" +} + +reconnect() { + pci_dev=${PCI_DEVS[$1]} + fn_dev=${FN_DEVS[$1]} + host_bridge=$(dirname $(dirname $(readlink -f $pci_dev))) + + # reconnect to prepare for surprise removal of the TSM or device + echo $(basename $tsm_link) > $pci_dev/tsm/connect + [[ $(cat $pci_dev/tsm/connect) == $(basename $tsm_link) ]] || err "$LINENO" + [[ -e $host_bridge/stream0.0.0 ]] || err "$LINENO" + [[ -e $tsm_link/$hb/stream0.0.0 ]] || err "$LINENO" +} + +devsec_test() { + setup_modules + + # find the tsm devices by personality + for tsm in /sys/class/tsm/tsm*; do + mode=$(cat $tsm/pci_mode) + [[ $mode == "devsec" ]] && tsm_devsec=$tsm + [[ $mode == "link" ]] && tsm_link=$tsm + done + [[ -n $tsm_devsec ]] || err "$LINENO" + [[ -n $tsm_link ]] || err "$LINENO" + + # check that devsec bus loads correctly and the TSM is detected + for i in ${!PCI_DEVS[@]}; do + pci_dev=${PCI_DEVS[$i]} + [[ -e $pci_dev ]] || err "$LINENO" + [[ -e $pci_dev/tsm ]] || err "$LINENO" + done + + ide_multi_test + ide_test 0 + tdisp_test 0 + + reconnect 0 + teardown_modules +} + +ORDER="bus" +devsec_test +ORDER="tsm" +devsec_test
MAINTAINERS +1 -0
@@ -26541,6 +26541,7 @@ F: include/linux/*tsm*.h F: include/uapi/linux/pci-tsm-netlink.h F: samples/devsec/ F: samples/tsm-mr/ +F: tools/testing/devsec/ TRUSTED SERVICES TEE DRIVER M: Balint Dobszay <[email protected]>
Signature · 1 line
2.52.0
PATCH v2 18/19 samples/devsec: Add evidence support
In reply to Dan WilliamsFor testing purposes add "certs" and "transcript" attributes to the devsec faux devices. Both the link_tsm and devsec_tsm reference the same shared data. The flow is:
- generate cert chain
- sign simulated evidence
- write blobs to "certs" and "transcript"
- trigger tsm/connect or tsm/lock to consume that evidence
--- samples/devsec/devsec.h | 5 ++ samples/devsec/common.c | 134 +++++++++++++++++++++++++++++++++++++- samples/devsec/link_tsm.c | 21 +++++- samples/devsec/tsm.c | 9 ++- 4 files changed, 166 insertions(+), 3 deletions(-)
samples/devsec/devsec.h +5 -0
@@ -40,4 +40,9 @@ static inline int devsec_get_domain_nr(struct devsec_sysdata *sd) }#endif extern struct devsec_sysdata *devsec_sysdata[NR_DEVSEC_HOST_BRIDGES ]; +extern const struct attribute_group devsec_evidence_group; +void devsec_evidence_busy (void ); +void devsec_evidence_idle (void ); +struct pci_tsm_evidence ; +void devsec_init_evidence (struct pci_tsm_evidence *evidence);#endif /* __DEVSEC_H__ */
samples/devsec/common.c +133 -1
155 lines not on this page · raw message
samples/devsec/link_tsm.c +20 -1
@@ -3,6 +3,7 @@#define dev_fmt (fmt) "devsec: " fmt#include <linux/device/faux.h> +#include <crypto/hash_info.h> #include <linux/pci-tsm.h> #include <linux/pci-ide.h> #include <linux/module.h>
@@ -51,6 +52,8 @@ static struct device *pci_tsm_host(struct pci_dev *pdev)static struct pci_tsm *devsec_tsm_pf0_probe (struct tsm_dev *tsm_dev,struct pci_dev *pdev) { +struct pci_tsm_evidence *evidence; +struct pci_tsm *tsm;int rc;dev_dbg (tsm_dev->dev.parent,"%s\n" ,pci_name (pdev));
@@ -60,10 +63,16 @@ static struct pci_tsm *devsec_tsm_pf0_probe(struct tsm_dev *tsm_dev,if (!devsec_tsm)return NULL ; + tsm = &devsec_tsm->pci.base_tsm; rc =pci_tsm_pf0_constructor (pdev, &devsec_tsm->pci, tsm_dev);if (rc)return NULL ; +devsec_evidence_busy (); + evidence = &tsm->evidence; +pci_tsm_init_evidence (evidence,0 ,HASH_ALGO_SHA384 ); +devsec_init_evidence (evidence); +pci_dbg (pdev,"TSM enabled\n" );return &no_free_ptr (devsec_tsm)->pci.base_tsm; }
@@ -113,6 +122,7 @@ static void devsec_link_tsm_pci_remove(struct pci_tsm *tsm)if (is_pci_tsm_pf0 (pdev)) {struct devsec_tsm_pf0 *devsec_tsm =to_devsec_tsm_pf0 (tsm); +devsec_evidence_idle ();pci_tsm_pf0_destructor (&devsec_tsm->pci);kfree (devsec_tsm); }else {
@@ -390,7 +400,16 @@ static struct attribute *devsec_link_attrs[] = { &dev_attr_tsm_request.attr, NULL, }; -ATTRIBUTE_GROUPS (devsec_link); + +static const struct attribute_group devsec_link_group = { + .attrs = devsec_link_attrs, +}; + +static const struct attribute_group *devsec_link_groups[] = { + &devsec_link_group, + &devsec_evidence_group, +NULL , +};static int __init devsec_link_tsm_init (void ) {
samples/devsec/tsm.c +8 -1
@@ -6,6 +6,7 @@#include <linux/pci-tsm.h> #include <linux/module.h> #include <linux/pci.h> +#include <linux/sysfs.h> #include <linux/tsm.h> #include "devsec.h"
@@ -105,9 +106,15 @@ static const struct faux_device_ops devsec_device_ops = { .probe = devsec_tsm_probe, }; +static const struct attribute_group *devsec_evidence_groups[] = { + &devsec_evidence_group, +NULL , +}; +static int __init devsec_tsm_init (void ) { - devsec_tsm =faux_device_create ("devsec_tsm" ,NULL , &devsec_device_ops); + devsec_tsm =faux_device_create_with_groups ( +"devsec_tsm" ,NULL , &devsec_device_ops, devsec_evidence_groups);if (!devsec_tsm)return -ENOMEM ;return 0 ;
Signature · 1 line
2.52.0
PATCH v2 19/19 tools/testing/devsec: Add basic evidence retrieval validation
In reply to Dan WilliamsCheckout basic operation of the pci-tsm-netlink ABI. The main complexity is reassembly of single evidence payloads that span multiple messages.
--- tools/testing/devsec/devsec.sh | 46 ++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+)
tools/testing/devsec/devsec.sh +46 -0
@@ -126,6 +126,46 @@ ide_multi_test() { done } +check_evidence() { + pci_dev=$1 + + set +x + + python3 tools/net/ynl/pyynl/cli.py --family pci-tsm --dump evidence-read \ + --json "{\"type-mask\": 2047, \"dev-name\": \"$(basename $pci_dev)\", \"flags\": 0}" \ + --output-json > json + + # Coalesce multi-message payloads where the protocol is a tuple + # of (type, val) followed by one more (val) only messages. + objects=() + for obj in $(jq -c '.[]' json); do + if [[ $(echo $obj | jq -r 'has("type")') == "true" ]]; then + t=$(echo $obj | jq -r '.type') + val_len=$(echo $obj | jq -r '.val | length') + objects[$t]=$val_len + else + val_len=$(echo $obj | jq -r '.val | length') + objects[$t]=$((objects[$t] + val_len)) + fi + done + + # Check that all 11 objects (PCI_TSM_EVIDENCE_TYPE_MAX) were + # returned and only objects 0 and 9 + # (PCI_TSM_EVIDENCE_TYPE_CERT0, + # PCI_TSM_EVIDENCE_TYPE_MEASUREMENTS) have a length of 8192 and + # the rest are empty. + [[ ${#objects[@]} -eq 11 ]] || err "$LINENO" + for i in ${!objects[@]}; do + if [[ $i == 0 || $i == 9 ]]; then + [[ ${objects[$i]} == 8192 ]] || err "$LINENO" + else + [[ ${objects[$i]} == 0 ]] || err "$LINENO" + fi + done + + set -x +} + ide_test() { pci_dev=${PCI_DEVS[$1]} fn_dev=${FN_DEVS[$1]}
@@ -155,6 +195,8 @@ ide_test() { dsm=$(cat $fn_dev/tsm/dsm) [[ $dsm == $(basename $pci_dev) ]] || err "$LINENO" + check_evidence $pci_dev + # bind both functions and validate that they display bound to # the TSM device echo $(basename $pci_dev) > $tsm_link/device/tsm_bind
@@ -213,6 +255,10 @@ devsec_test() { [[ -n $tsm_devsec ]] || err "$LINENO" [[ -n $tsm_link ]] || err "$LINENO" + # initialize evidence payloads + dd if=/dev/zero of=/sys/bus/faux/devices/devsec_link_tsm/certs bs=4K count=1 + dd if=/dev/zero of=/sys/bus/faux/devices/devsec_link_tsm/transcript bs=4K count=1 + # check that devsec bus loads correctly and the TSM is detected for i in ${!PCI_DEVS[@]}; do pci_dev=${PCI_DEVS[$i]}
Signature · 1 line
2.52.0
Aneesh Kumar K.V
maintainerDan Williams <[email protected]> writes:
....
To support devices without IDE/DOE support we need something similar.
modified drivers/pci/tsm/core.c
@@ -1236,12 +1236,14 @@ int pci_tsm_pf0_constructor(struct pci_dev *pdev, struct pci_tsm_pf0 *tsm,
struct tsm_dev *tsm_dev)
{
mutex_init(&tsm->lock);
- tsm->doe_mb = pci_find_doe_mailbox(pdev, PCI_VENDOR_ID_PCI_SIG,
- PCI_DOE_FEATURE_CMA);
- if (!tsm->doe_mb) {
- pci_warn(pdev, "TSM init failure, no CMA mailbox\n");
- return -ENODEV;
- }
+
+ /*
+ * Note, low-level TSM driver responsible for determining if it wants to + * proceed with a device that has no DOE mailbox. TSM may have an + * alternate method for coordinating TDISP. + */
+ if (!tsm->doe_mb)
+ pci_dbg(pdev, "no CMA mailbox\n");
return pci_tsm_link_constructor(pdev, &tsm->base_tsm, tsm_dev);
}
Quoted 15 lines · Aneesh Kumar K.V
Dan Williams <[email protected]> writes:
....
To support devices without IDE/DOE support we need something similar.
modified
drivers/pci/tsm/core.c(quoted diff) +0
-4@@ -1236,12 +1236,14 @@ int pci_tsm_pf0_constructor(struct pci_dev *pdev, struct pci_tsm_pf0 *tsm, struct tsm_dev *tsm_dev) { mutex_init(&tsm->lock); - tsm->doe_mb = pci_find_doe_mailbox(pdev, PCI_VENDOR_ID_PCI_SIG, - PCI_DOE_FEATURE_CMA); - if (!tsm->doe_mb) { - pci_warn(pdev, "TSM init failure, no CMA mailbox\n");
Review int pci_tsm_pf0_constructor(struct pci_dev *pdev, struct pci_tsm_pf0 *tsm, · line 1246
Quoted 10 lines
(quoted diff) +8
-2- return -ENODEV; - } + + /* + * Note, low-level TSM driver responsible for determining if it wants to + * proceed with a device that has no DOE mailbox. TSM may have an + * alternate method for coordinating TDISP. + */ + if (!tsm->doe_mb) + pci_dbg(pdev, "no CMA mailbox\n");
A patch like patch can go upstream now. Care to send?