Skip to contentSkip to conversation

ARM CCA Device Assignment support

RFC PATCH In review Aneesh Kumar K.V (Arm) 14 participants 188 messages Jul 28, 2025 – Oct 15, 2025 Page 2 of 2

Subsystems (13)
KVM PCI Crypto API arm64 +9 more subsystems
lore
Share & tools
lore
#151 lore raw

Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report

In reply to Arto Merilainen
Quoted 15 lines · Arto Merilainen
Quoted 13 lines · Aneesh Kumar K.V (Arm)
drivers/virt/coco/arm-cca-guest/rsi-da.c +13 -0
+	for (int i = 0; i < interface_report->mmio_range_count; i++, mmio_range++) {
+
+		/*FIXME!! units in 4K size*/
+		range_id = FIELD_GET(TSM_INTF_REPORT_MMIO_RANGE_ID, mmio_range->range_attributes);
+
+		/* no secure interrupts */
+		if (msix_tbl_bar != -1 && range_id == msix_tbl_bar) {
+			pr_info("Skipping misx table\n");
+			continue;
+		}
+
+		if (msix_pba_bar != -1 && range_id == msix_pba_bar) {
+			pr_info("Skipping misx pba\n");
Quoted 8 lines

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 292

Quoted 3 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +3 -0
+			continue;
+		}
+

MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated.

Right, there are not enough BARs in most devices to give MSI its own BAR.

Quoted 4 lines

It is enough to have sufficient isolation within the
BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.

Right, this sounds bad.

Quoted 3 lines

Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?

IMHO this is a mess. The virtualization must end up putting a shared page(s) covering the MSI space in the middle of the MMIO region.

I think this should be done by fragmenting the layout in the IPA where the private MMIO is within the protected IPA space with an unmapped hole covering the MSIX registers. The acceptance process should validate this.

The MSIX registers would then be located in the shared IPA space.

A normal driver mmaping it's BAR will then crash if it tries to access the MSIX registers. This is good, we want to catch these non-secure configurations and block them.

The MSI code will have to know to compute the shared IPA alias and use that.

Jason

author maintainer
#152 lore raw

Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report

In reply to Arto Merilainen
Quoted 27 lines · Arto Merilainen
Quoted 6 lines · Aneesh Kumar K.V (Arm)
drivers/virt/coco/arm-cca-guest/rsi-da.c +6 -0
+	for (int i = 0; i < interface_report->mmio_range_count; i++, mmio_range++) {
+
+		/*FIXME!! units in 4K size*/
+		range_id = FIELD_GET(TSM_INTF_REPORT_MMIO_RANGE_ID, mmio_range->range_attributes);
+
+		/* no secure interrupts */

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 297

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+		if (msix_tbl_bar != -1 && range_id == msix_tbl_bar) {
+			pr_info("Skipping misx table\n");
+			continue;
+		}
+
+		if (msix_pba_bar != -1 && range_id == msix_pba_bar) {
+			pr_info("Skipping misx pba\n");
+			continue;
+		}
+

MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated. It is enough to have sufficient isolation within the BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.

Quoted 4 lines

Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?

Yes, that was added because at one point the FVP model was including the MSI-X table and PBA regions in the interface report.

If I understand correctly, we shouldn't expect to see those regions in the report unless secure interrupts are supported. The BAR-based skipping was added as a workaround to handle the FVP issue.

I believe we can drop that workaround now—if those regions are incorrectly present, the below validation logic should catch and reject them appropriately. Does that sound reasonable?

		/* No secure interrupts, we should not find this set, ignore for now. */
		if (FIELD_GET(TSM_INTF_REPORT_MMIO_MSIX_TABLE, mmio_range->range_attributes) ||
		    FIELD_GET(TSM_INTF_REPORT_MMIO_PBA, mmio_range->range_attributes)) {
			pci_info(pdev, "Skipping MSIX (%ld/%ld) range [%d] #%d %d pages, %llx..%llx\n",
				 FIELD_GET(TSM_INTF_REPORT_MMIO_MSIX_TABLE, mmio_range->range_attributes),
				 FIELD_GET(TSM_INTF_REPORT_MMIO_PBA, mmio_range->range_attributes),
				 i, range_id, mmio_range->num_pages, r->start, r->end);
			continue;
		}

-aneesh

#153 lore raw

Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report

Quoted 29 lines · Aneesh Kumar K.V
Quoted 27 lines · Arto Merilainen
Quoted 6 lines · Aneesh Kumar K.V (Arm)
drivers/virt/coco/arm-cca-guest/rsi-da.c +6 -0
+    for (int i = 0; i < interface_report->mmio_range_count; i++, mmio_range++) {
+
+            /*FIXME!! units in 4K size*/
+            range_id = FIELD_GET(TSM_INTF_REPORT_MMIO_RANGE_ID, mmio_range->range_attributes);
+
+            /* no secure interrupts */

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 297

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+            if (msix_tbl_bar != -1 && range_id == msix_tbl_bar) {
+                    pr_info("Skipping misx table\n");
+                    continue;
+            }
+
+            if (msix_pba_bar != -1 && range_id == msix_pba_bar) {
+                    pr_info("Skipping misx pba\n");
+                    continue;
+            }
+

MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated. It is enough to have sufficient isolation within the BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.

Quoted 7 lines
Quoted 4 lines

Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?

Yes, that was added because at one point the FVP model was including the MSI-X table and PBA regions in the interface report.

The issue I am raising is a different one. The MSI-X table and PBA may reside in the middle of a BAR range, and the BAR range may contain registers which should be accessible only via protected IPA. In this case I would expect the pages before and after the MSI-X table (and PBA) to be validated while the pages related to MSI-X structures would be left unprotected.

Given that the MSI-X table or PBA regions shouldn't be present in the interface report, the code needs to first find out where the MSI-X structures reside, and use this information to determine which "sub-ranges" should be skipped over during validation.

Quoted 4 lines

If I understand correctly, we shouldn't expect to see those regions in the report unless secure interrupts are supported. The BAR-based skipping was added as a workaround to handle the FVP issue.

Correct. Assuming that RMM doesn't lock the MSI-X table, I'd expect these regions to be omitted in the interface report.

Quoted 4 lines

I believe we can drop that workaround now—if those regions are incorrectly present, the below validation logic should catch and reject them appropriately. Does that sound reasonable?

Quoted 11 lines
                 /* No secure interrupts, we should not find this set, ignore for now. */
                 if (FIELD_GET(TSM_INTF_REPORT_MMIO_MSIX_TABLE, mmio_range->range_attributes) ||
                     FIELD_GET(TSM_INTF_REPORT_MMIO_PBA, mmio_range->range_attributes)) {
                         pci_info(pdev, "Skipping MSIX (%ld/%ld) range [%d] #%d %d pages, %llx..%llx\n",
                                  FIELD_GET(TSM_INTF_REPORT_MMIO_MSIX_TABLE, mmio_range->range_attributes),
                                  FIELD_GET(TSM_INTF_REPORT_MMIO_PBA, mmio_range->range_attributes),
                                  i, range_id, mmio_range->num_pages, r->start, r->end);
                         continue;
                 }

First, I think this is skipping over the whole range => if there are registers outside of the MSI-X structures (but within the same BAR), they won't be validated.

Second, if the MSI-X regions are present in the interface report, wouldn't it - in the common case - mean that the device expects the ranges to be accessed with T=1? If this happens unexpectedly, it sounds that MSI-X wouldn't be usable. I wonder if the code should simply return an error instead if informing the user via pci_info()...

- R2

#154 lore raw

Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report

In reply to Arto Merilainen
Quoted 24 lines · Arto Merilainen
Quoted 8 lines · Aneesh Kumar K.V (Arm)

+    for (int i = 0; i < interface_report->mmio_range_count; i++, mmio_range++) {
+
+        /*FIXME!! units in 4K size*/
+        range_id = FIELD_GET(TSM_INTF_REPORT_MMIO_RANGE_ID, mmio_range->range_attributes);

drivers/virt/coco/arm-cca-guest/rsi-da.c +2 -0
+
+        /* no secure interrupts */

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 297

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+        if (msix_tbl_bar != -1 && range_id == msix_tbl_bar) {
+            pr_info("Skipping misx table\n");
+            continue;
+        }
+
+        if (msix_pba_bar != -1 && range_id == msix_pba_bar) {
+            pr_info("Skipping misx pba\n");
+            continue;
+        }
+

MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated. It is enough to have sufficient isolation within the BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.

Quoted 2 lines

Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?

I re-reviewed my suggestion, and what I proposed here seems wrong. However, I think there is a different more generic problem related to the MSI-X table, PBA and non-TEE ranges.

If a BAR is sparse (e.g., it has TEE pages and the MSI-X table, PBA or non-TEE areas), the TDISP interface report may contain multiple ranges with the same range_id (/BAR id). In case a BAR contains some registers in low addresses, the MSI-X table and other registers after the MSI-X table, the interface report is expected to have two ranges for the same BAR with different "first 4k page" and "size" fields.

This creates a tricky problem given that RSI_VDEV_VALIDATE_MAPPING requires both the ipa_base and pa_base which should correspond to the same location. In above scenario, the PA of the first range would correspond to the BAR base whereas the second range would correspond to a location residing after the MSI-X table.

Assuming that the report contains obfuscated (but linear) physical addresses, it would be possible to create heuristics for this case. However, the fundamental problem is that none of the "first 4k page" fields in the ranges is guaranteed to correspond to the base of any BAR: Consider a case where the MSI-X table is in the beginning of a BAR and it is followed by a single TEE range. If the MSI-X is not locked, the "first 4k page" field will not correspond to the beginning of the BAR. If the realm naiviely reads the ipa_base using pci_resouce_n() and corresponding pa_base from the interface report, the addresses won't match and the validation will fail.

It seems that interpreting the interface report cannot be done without knowledge of the device's register layout. Therefore, I don't think the ranges can be validated/remapped automatically without involving the device driver, but there should be APIs for reading the interface report, and for requesting making specific ranges protected.

- R2

#155 lore raw

Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report

Quoted 5 lines · Arto Merilainen

This creates a tricky problem given that RSI_VDEV_VALIDATE_MAPPING requires both the ipa_base and pa_base which should correspond to the same location. In above scenario, the PA of the first range would correspond to the BAR base whereas the second range would correspond to a location residing after the MSI-X table.

This seems like a defect in the RSI_VDEV_VALIDATE_MAPPING - it should be able to consume the same format of data that the tdisp report emits to validate it.

From a kernel side we also should be careful that the driver isn't tricked into mapping MMIO that is not secure when it should be. Presumably all the default io access functions should demand secure memory in T=1 mode, and special ones like the MSI-X code would have some special version to accept either?

Jason

author maintainer
#156 lore raw

Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report

In reply to Arto Merilainen
Quoted 62 lines · Arto Merilainen
Quoted 29 lines · Arto Merilainen
Quoted 8 lines · Aneesh Kumar K.V (Arm)

+    for (int i = 0; i < interface_report->mmio_range_count; i++, mmio_range++) {
+
+        /*FIXME!! units in 4K size*/
+        range_id = FIELD_GET(TSM_INTF_REPORT_MMIO_RANGE_ID, mmio_range->range_attributes);

drivers/virt/coco/arm-cca-guest/rsi-da.c +2 -0
+
+        /* no secure interrupts */

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 297

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+        if (msix_tbl_bar != -1 && range_id == msix_tbl_bar) {
+            pr_info("Skipping misx table\n");
+            continue;
+        }
+
+        if (msix_pba_bar != -1 && range_id == msix_pba_bar) {
+            pr_info("Skipping misx pba\n");
+            continue;
+        }
+

MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated. It is enough to have sufficient isolation within the BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.

Quoted 3 lines

Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?

I re-reviewed my suggestion, and what I proposed here seems wrong. However, I think there is a different more generic problem related to the MSI-X table, PBA and non-TEE ranges.

If a BAR is sparse (e.g., it has TEE pages and the MSI-X table, PBA or non-TEE areas), the TDISP interface report may contain multiple ranges with the same range_id (/BAR id). In case a BAR contains some registers in low addresses, the MSI-X table and other registers after the MSI-X table, the interface report is expected to have two ranges for the same BAR with different "first 4k page" and "size" fields.

This creates a tricky problem given that RSI_VDEV_VALIDATE_MAPPING requires both the ipa_base and pa_base which should correspond to the same location. In above scenario, the PA of the first range would correspond to the BAR base whereas the second range would correspond to a location residing after the MSI-X table.

Assuming that the report contains obfuscated (but linear) physical addresses, it would be possible to create heuristics for this case. However, the fundamental problem is that none of the "first 4k page" fields in the ranges is guaranteed to correspond to the base of any BAR:
Consider a case where the MSI-X table is in the beginning of a BAR and it is followed by a single TEE range. If the MSI-X is not locked, the "first 4k page" field will not correspond to the beginning of the BAR. If the realm naiviely reads the ipa_base using pci_resouce_n() and corresponding pa_base from the interface report, the addresses won't match and the validation will fail.

Quoted 6 lines

It seems that interpreting the interface report cannot be done without knowledge of the device's register layout. Therefore, I don't think the ranges can be validated/remapped automatically without involving the device driver, but there should be APIs for reading the interface report, and for requesting making specific ranges protected.

But we need to validate the interface report before accepting the device, and the device driver is only loaded after the device has been accepted.

Can we assume that only the MSI-X table and PBA ranges may be missing from the interface report, while all other non-secure regions are reported as NON-TEE ranges?

If so, we could retrieve the MSI-X guest real address details from config space and map the beginning of the BAR correctly.

Dan / Yilun — how is this handled in Intel TDX?

From what I can see, the AMD patches appear to encounter the same issue.

-aneesh

#157 lore raw

Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report

Quoted 71 lines · Aneesh Kumar K.V
Quoted 62 lines · Arto Merilainen
Quoted 29 lines · Arto Merilainen
Quoted 8 lines · Aneesh Kumar K.V (Arm)

+    for (int i = 0; i < interface_report->mmio_range_count; i++, mmio_range++) {
+
+        /*FIXME!! units in 4K size*/
+        range_id = FIELD_GET(TSM_INTF_REPORT_MMIO_RANGE_ID, mmio_range->range_attributes);

drivers/virt/coco/arm-cca-guest/rsi-da.c +2 -0
+
+        /* no secure interrupts */

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 297

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+        if (msix_tbl_bar != -1 && range_id == msix_tbl_bar) {
+            pr_info("Skipping misx table\n");
+            continue;
+        }
+
+        if (msix_pba_bar != -1 && range_id == msix_pba_bar) {
+            pr_info("Skipping misx pba\n");
+            continue;
+        }
+

MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated. It is enough to have sufficient isolation within the BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.

Quoted 3 lines

Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?

I re-reviewed my suggestion, and what I proposed here seems wrong. However, I think there is a different more generic problem related to the MSI-X table, PBA and non-TEE ranges.

If a BAR is sparse (e.g., it has TEE pages and the MSI-X table, PBA or non-TEE areas), the TDISP interface report may contain multiple ranges with the same range_id (/BAR id). In case a BAR contains some registers in low addresses, the MSI-X table and other registers after the MSI-X table, the interface report is expected to have two ranges for the same BAR with different "first 4k page" and "size" fields.

This creates a tricky problem given that RSI_VDEV_VALIDATE_MAPPING requires both the ipa_base and pa_base which should correspond to the same location. In above scenario, the PA of the first range would correspond to the BAR base whereas the second range would correspond to a location residing after the MSI-X table.

Assuming that the report contains obfuscated (but linear) physical addresses, it would be possible to create heuristics for this case. However, the fundamental problem is that none of the "first 4k page" fields in the ranges is guaranteed to correspond to the base of any BAR:
Consider a case where the MSI-X table is in the beginning of a BAR and it is followed by a single TEE range. If the MSI-X is not locked, the "first 4k page" field will not correspond to the beginning of the BAR. If the realm naiviely reads the ipa_base using pci_resouce_n() and corresponding pa_base from the interface report, the addresses won't match and the validation will fail.

Quoted 6 lines

It seems that interpreting the interface report cannot be done without knowledge of the device's register layout. Therefore, I don't think the ranges can be validated/remapped automatically without involving the device driver, but there should be APIs for reading the interface report, and for requesting making specific ranges protected.

Quoted 6 lines

But we need to validate the interface report before accepting the device, and the device driver is only loaded after the device has been accepted.

Can we assume that only the MSI-X table and PBA ranges may be missing from the interface report, while all other non-secure regions are reported as NON-TEE ranges?

In case of the "some registers"+MSIX+"some more registers" BAR#X, I'd rather expect 3 ranges in the report and not just 2, for each subrange, sorted, with/without NonTEE bit set, as the PCIe spec suggests (sort of, I struggle parsing it):

===
When reporting the MMIO range for a TDI, the MMIO ranges must be reported in the logical order in which the TDI MMIO range is configured such that the first range reported corresponds to first range of pages in the TDI and so on
===

And if the number of pages in the report differs from the BAR size, then we should fail validation. Otherwise it is impossible to tell from the report's MMIO addresses what part of a BAR needs to be validated (==TEE) and if the guest device driver has to know it - then reports are just useless (which is hardly true).

Quoted 6 lines

If so, we could retrieve the MSI-X guest real address details from config space and map the beginning of the BAR correctly.

Dan / Yilun — how is this handled in Intel TDX?

From what I can see, the AMD patches appear to encounter the same issue.

I am skipping MSIX BAR because T=1 is tied to C=1 so after such validation, the BAR belongs to the guest and the hw will reject VFIO-PCI (==HV) attempts to write to MSIX BAR. Probably too straight forward though and I can try assigning it to the host (and add Cbit in those PTEs), and see how the PSP handles it (not). Thanks,

Quoted 2 lines

-aneesh

Signature · 1 line
Alexey
#158 lore raw

Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report

In reply to Aneesh Kumar K.V
Quoted 2 lines · Aneesh Kumar K.V

But we need to validate the interface report before accepting the device, and the device driver is only loaded after the device has been accepted.

+1

This must work from the generic OS code.

So I'd say add a new TSM op:
int validate_pci_bar_range(struct pci_dev *pdev,
unsigned int bar_index, u64 tdisp_pa, u64 size,phys_addr_t *bar_offset_out);

TSM has broadly two options to compute bar_offset_out:

1) Require the TDISP MMIO Offset is aligned to the BAR size and use something like:

    *bar_offset_out = (tdisp_pa) % pci_resource_len(pdev, bar_index);
    ipa = pci_resource_start(pdev, bar_index) + *bar_offset_out;
    if (size + *bar_offset_out > pci_resource_len(pdev, bar_index))
        return -EINVAL;
    tsm_call_to_validate(pdev, ipa, pa, size)

2) Require the TSM to convert the offest'd PA to the IPA:

    tsm_call_to_convert(pdev, pa, size, &ipa);

    if (ipa < pci_resource_start(pdev, bar_index) ||
        ipa >= pci_resource_end(pdev, bar_index) ||
        (ipa + size) > pci_resource_end(pdev, bar_index))
	return -EINVAL;

    *bar_offset_out = ipa -  pci_resource_start(pdev, bar_index);

Then the generic code builds a map of what parts of the BAR are secure and what are not.

If it can't do either the TSM is unusable by Linux.

Jason

#159 lore raw

Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report

In reply to Aneesh Kumar K.V
Quoted 78 lines · Aneesh Kumar K.V
Quoted 62 lines · Arto Merilainen
Quoted 29 lines · Arto Merilainen
Quoted 8 lines · Aneesh Kumar K.V (Arm)

+    for (int i = 0; i < interface_report->mmio_range_count; i++, mmio_range++) {
+
+        /*FIXME!! units in 4K size*/
+        range_id = FIELD_GET(TSM_INTF_REPORT_MMIO_RANGE_ID, mmio_range->range_attributes);

drivers/virt/coco/arm-cca-guest/rsi-da.c +2 -0
+
+        /* no secure interrupts */

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 297

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+        if (msix_tbl_bar != -1 && range_id == msix_tbl_bar) {
+            pr_info("Skipping misx table\n");
+            continue;
+        }
+
+        if (msix_pba_bar != -1 && range_id == msix_pba_bar) {
+            pr_info("Skipping misx pba\n");
+            continue;
+        }
+

MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated. It is enough to have sufficient isolation within the BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.

Quoted 3 lines

Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?

I re-reviewed my suggestion, and what I proposed here seems wrong. However, I think there is a different more generic problem related to the MSI-X table, PBA and non-TEE ranges.

If a BAR is sparse (e.g., it has TEE pages and the MSI-X table, PBA or non-TEE areas), the TDISP interface report may contain multiple ranges with the same range_id (/BAR id). In case a BAR contains some registers in low addresses, the MSI-X table and other registers after the MSI-X table, the interface report is expected to have two ranges for the same BAR with different "first 4k page" and "size" fields.

This creates a tricky problem given that RSI_VDEV_VALIDATE_MAPPING requires both the ipa_base and pa_base which should correspond to the same location. In above scenario, the PA of the first range would correspond to the BAR base whereas the second range would correspond to a location residing after the MSI-X table.

Assuming that the report contains obfuscated (but linear) physical addresses, it would be possible to create heuristics for this case. However, the fundamental problem is that none of the "first 4k page" fields in the ranges is guaranteed to correspond to the base of any BAR:
Consider a case where the MSI-X table is in the beginning of a BAR and it is followed by a single TEE range. If the MSI-X is not locked, the "first 4k page" field will not correspond to the beginning of the BAR. If the realm naiviely reads the ipa_base using pci_resouce_n() and corresponding pa_base from the interface report, the addresses won't match and the validation will fail.

Quoted 6 lines

It seems that interpreting the interface report cannot be done without knowledge of the device's register layout. Therefore, I don't think the ranges can be validated/remapped automatically without involving the device driver, but there should be APIs for reading the interface report, and for requesting making specific ranges protected.

But we need to validate the interface report before accepting the device, and the device driver is only loaded after the device has been accepted.

Can we assume that only the MSI-X table and PBA ranges may be missing from the interface report, while all other non-secure regions are reported as NON-TEE ranges?

Quoted 6 lines

If so, we could retrieve the MSI-X guest real address details from config space and map the beginning of the BAR correctly.

Dan / Yilun — how is this handled in Intel TDX?

From what I can see, the AMD patches appear to encounter the same issue.

Same issue exists for TDX. In the near term this solidifies that the PCI/TSM core should not be assumining anything with respect to marking MMIO ranges as private, and leave that all the to low-level TSM driver.

...but then yes I expect we need to build some common infrastructure for special casing MSIX.

#160 lore raw

Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report

In reply to Aneesh Kumar K.V (Arm)
Quoted 20 lines · Aneesh Kumar K.V (Arm)

This starts the sequence to transition the realm device to the TDISP RUN state by writing 1 to 'tsm/accept'.

+++ b/drivers/virt/coco/arm-cca-guest/rsi-da.c

drivers/virt/coco/arm-cca-guest/rsi-da.c +15 -0
+int rsi_update_interface_report(struct pci_dev *pdev, bool validate)
+{
+	int ret;
+	struct resource *r;
+	int msix_tbl_bar, msix_pba_bar;
+	unsigned int range_id;
+	unsigned long mmio_start_phys;
+	unsigned long mmio_flags = 0; /* non coherent, not limited order */
+	struct pci_tdisp_mmio_range *mmio_range;
+	struct pci_tdisp_device_interface_report *interface_report;
+	struct cca_guest_dsc *dsm = to_cca_guest_dsc(pdev);
+	int vdev_id = (pci_domain_nr(pdev->bus) << 16) |
+		PCI_DEVID(pdev->bus->number, pdev->devfn);
+
+

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 284

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+	interface_report = (struct pci_tdisp_device_interface_report *)dsm->interface_report;
+	mmio_range = (struct pci_tdisp_mmio_range *)(interface_report + 1);
+
+
+	msix_tbl_bar = get_msix_bar(pdev, PCI_MSIX_TABLE);
+	msix_pba_bar = get_msix_bar(pdev, PCI_MSIX_PBA);
+
+	for (int i = 0; i < interface_report->mmio_range_count; i++, mmio_range++) {
+
+		/*FIXME!! units in 4K size*/

I guess you intend to fix this?

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 294

Quoted 8 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +8 -0
+		/* no secure interrupts */
+		if (msix_tbl_bar != -1 && range_id == msix_tbl_bar) {
+			pr_info("Skipping misx table\n");
+			continue;
+		}
+
+		if (msix_pba_bar != -1 && range_id == msix_pba_bar) {
+			pr_info("Skipping misx pba\n");

s/misx/MSI-X/ (twice)
s/pba/PBA/

author maintainer
#161 lore raw

RFC PATCH 35/38 coco: guest: arm64: Add Realm device start and stop support

In reply to Aneesh Kumar K.V (Arm)

Writing 1 to 'tsm/acceept' will initiate the TDISP RUN sequence.

---
 arch/arm64/include/asm/rsi_cmds.h         | 19 +++++++
 arch/arm64/include/asm/rsi_smc.h          |  2 +
 drivers/virt/coco/arm-cca-guest/arm-cca.c | 15 ++++++
 drivers/virt/coco/arm-cca-guest/rsi-da.c  | 60 +++++++++++++++++++++++
 drivers/virt/coco/arm-cca-guest/rsi-da.h  |  2 +-
 5 files changed, 97 insertions(+), 1 deletion(-)
arch/arm64/include/asm/rsi_cmds.h +19 -0
index 1cc00d404e53..3463d571d7db 100644
@@ -246,4 +246,23 @@ rsi_rdev_validate_mapping(unsigned long vdev_id, unsigned long inst_id,
 		return -EINVAL;
 	return 0;
 }
+
+static inline unsigned long __rsi_rdev_start(unsigned long vdev_id, unsigned long inst_id)
+{
+	struct arm_smccc_res res;
+
+	arm_smccc_1_1_invoke(SMC_RSI_RDEV_START, vdev_id, inst_id, &res);
+
+	return res.a0;
+}
+
+static inline unsigned long __rsi_rdev_stop(unsigned long vdev_id, unsigned long inst_id)
+{
+	struct arm_smccc_res res;
+
+	arm_smccc_1_1_invoke(SMC_RSI_RDEV_STOP, vdev_id, inst_id, &res);
+
+	return res.a0;
+}
+
 #endif /* __ASM_RSI_CMDS_H */
arch/arm64/include/asm/rsi_smc.h +2 -0
index a28b41cf01ca..f6aa647239c0 100644
@@ -203,6 +203,8 @@ struct rsi_host_call {
 #define SMC_RSI_RDEV_GET_INTERFACE_REPORT	SMC_RSI_FID(0x1a6)
 
 #define SMC_RSI_RDEV_LOCK			SMC_RSI_FID(0x1a9)
+#define SMC_RSI_RDEV_START			SMC_RSI_FID(0x1aa)
+#define SMC_RSI_RDEV_STOP			SMC_RSI_FID(0x1ab)
 
 #define RSI_DEV_MEM_COHERENT		BIT(0)
 #define RSI_DEV_MEM_LIMITED_ORDER	BIT(1)
drivers/virt/coco/arm-cca-guest/arm-cca.c +15 -0
index c1cefb983ac7..7eeb9732d20a 100644
@@ -250,6 +250,8 @@ static void cca_tsm_unlock(struct pci_dev *pdev)
 	/* invalidate dev mapping based on interface report */
 	rsi_update_interface_report(pdev, false);
 
+	rsi_device_stop(pdev);
+
 	ret = rhi_da_vdev_set_tdi_state(vdev_id, RHI_DA_TDI_CONFIG_UNLOCKED);
 	if (ret) {
 		pci_err(pdev, "failed to TSM unbind the device (%ld)\n", ret);
@@ -257,11 +259,24 @@ static void cca_tsm_unlock(struct pci_dev *pdev)
 	}
 }
 
+static int cca_tsm_accept(struct pci_dev *pdev)
+{
+	int ret;
+
+	ret = rsi_device_start(pdev);
+	if (ret) {
+		pci_err(pdev, "failed to transition the device to run state (%d)\n", ret);
+		return -EIO;
+	}
+	return 0;
+}
+
 static const struct pci_tsm_ops cca_pci_ops = {
 	.probe = cca_tsm_pci_probe,
 	.remove = cca_tsm_pci_remove,
 	.lock = cca_tsm_lock,
 	.unlock = cca_tsm_unlock,
+	.accept	 = cca_tsm_accept,
 };
 
 static void cca_tsm_unregister(void *tsm)
drivers/virt/coco/arm-cca-guest/rsi-da.c +60 -0

77 lines not on this page · raw message

drivers/virt/coco/arm-cca-guest/rsi-da.h +1 -1
index 0d6e1c0ada4a..71ee1edb832e 100644
@@ -54,5 +54,5 @@ static inline struct cca_guest_dsc *to_cca_guest_dsc(struct pci_dev *pdev)
 int rsi_update_interface_report(struct pci_dev *pdev, bool validate);
 int rsi_device_lock(struct pci_dev *pdev);
 int rsi_device_start(struct pci_dev *pdev);
-
+int rsi_device_stop(struct pci_dev *pdev);
 #endif
Signature · 1 line
2.43.0
#162 lore raw

Re: [RFC PATCH v1 35/38] coco: guest: arm64: Add Realm device start and stop support

Quoted 3 lines · On Mon, 28 Jul 2025 19:22:12 +0530 "Aneesh Kumar K.V (Arm)"…

Writing 1 to 'tsm/acceept' will initiate the TDISP RUN sequence.

A few more trivial things on this first read through.

Jonathan

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_start(struct pci_dev *pdev) · line 401

Quoted 11 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +6 -0
index 936f844880de..64034d220e02 100644
 
+int rsi_device_stop(struct pci_dev *pdev)
+{
+	int ret;
+	struct cca_guest_dsc *dsm = to_cca_guest_dsc(pdev);
+	int vdev_id = (pci_domain_nr(pdev->bus) << 16) |
+		PCI_DEVID(pdev->bus->number, pdev->devfn);

Feels like this occurs so often we should add a helper. Can't be completely generic as pci_domain_nr can have more bits than I guess we are assuming here, but we can have something for use in the rsi code.

Quoted 13 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +6 -0
+
+	ret = rsi_rdev_stop(pdev, vdev_id, dsm->instance_id);
+	if (ret != RSI_SUCCESS) {
+		pci_err(pdev, "failed to stop the device (%u)\n", ret);
+		return -EIO;
+	}
 	return 0;
 }
drivers/virt/coco/arm-cca-guest/rsi-da.h +0 -0
index 0d6e1c0ada4a..71ee1edb832e 100644

drivers/virt/coco/arm-cca-guest/rsi-da.h static inline struct cca_guest_dsc *to_cca_guest_dsc(struct pci_dev *pdev) · line 57

Quoted 4 lines
drivers/virt/coco/arm-cca-guest/rsi-da.h +0 -1
@@ -54,5 +54,5 @@ static inline struct cca_guest_dsc *to_cca_guest_dsc(struct pci_dev *pdev)
 int rsi_update_interface_report(struct pci_dev *pdev, bool validate);
 int rsi_device_lock(struct pci_dev *pdev);
 int rsi_device_start(struct pci_dev *pdev);
-

Tidy this up.

Quoted 2 lines
drivers/virt/coco/arm-cca-guest/rsi-da.h +1 -0
+int rsi_device_stop(struct pci_dev *pdev);
 #endif
#163 lore raw

Re: [RFC PATCH v1 35/38] coco: guest: arm64: Add Realm device start and stop support

In reply to Aneesh Kumar K.V (Arm)
Quoted 1 line · Aneesh Kumar K.V (Arm)

Writing 1 to 'tsm/acceept' will initiate the TDISP RUN sequence.

s/acceept/accept/

author maintainer
#164 lore raw

RFC PATCH 36/38 KVM: arm64: CCA: enable DA in realm create parameters

In reply to Aneesh Kumar K.V (Arm)

Now that we have all the required steps for DA in-place, enable DA while creating ralm.

---
 arch/arm64/include/asm/rmi_smc.h | 1 +
 arch/arm64/kvm/rme.c             | 2 ++
 2 files changed, 3 insertions(+)
arch/arm64/include/asm/rmi_smc.h +1 -0
index ab169b375198..f664954a2a91 100644
@@ -112,6 +112,7 @@ enum rmi_ripas {
 #define RMI_REALM_PARAM_FLAG_LPA2		BIT(0)
 #define RMI_REALM_PARAM_FLAG_SVE		BIT(1)
 #define RMI_REALM_PARAM_FLAG_PMU		BIT(2)
+#define RMI_REALM_PARAM_FLAG_DA			BIT(3)
 
 /*
  * Note many of these fields are smaller than u64 but all fields have u64
arch/arm64/kvm/rme.c +2 -0
index 11c8d47e3e9b..394d1534e6c2 100644
@@ -695,6 +695,8 @@ static int realm_create_rd(struct kvm *kvm)
 	if (r)
 		goto out_undelegate_tables;
 
+	/* For now default enable DA */
+	params->flags = RMI_REALM_PARAM_FLAG_DA;
 	params_phys = virt_to_phys(params);
 
 	if (rmi_realm_create(rd_phys, params_phys)) {
Signature · 1 line
2.43.0
#165 lore raw

Re: [RFC PATCH v1 36/38] KVM: arm64: CCA: enable DA in realm create parameters

Quoted 2 lines · Aneesh Kumar K.V (Arm)

Now that we have all the required steps for DA in-place, enable DA while creating ralm.

s/ralm/realm/ ?

author maintainer
#166 lore raw

RFC PATCH 37/38 coco: guest: arm64: Add support for fetching device measurements

In reply to Aneesh Kumar K.V (Arm)

Fetch device measurements using RSI_RDEV_GET_MEASUREMENTS.

---
 arch/arm64/include/asm/rsi_cmds.h        | 11 +++++++
 arch/arm64/include/asm/rsi_smc.h         | 16 ++++++++++
 drivers/virt/coco/arm-cca-guest/rsi-da.c | 39 ++++++++++++++++++++++++
 drivers/virt/coco/arm-cca-guest/rsi-da.h |  2 ++
 4 files changed, 68 insertions(+)
arch/arm64/include/asm/rsi_cmds.h +11 -0
index 3463d571d7db..42b998f44a0e 100644
@@ -265,4 +265,15 @@ static inline unsigned long __rsi_rdev_stop(unsigned long vdev_id, unsigned long
 	return res.a0;
 }
 
+static inline unsigned long __rsi_rdev_get_measurements(unsigned long vdev_id,
+						       unsigned long inst_id,
+						       phys_addr_t meas)
+{
+	struct arm_smccc_res res;
+
+	arm_smccc_1_1_invoke(SMC_RSI_RDEV_GET_MEASUREMENTS, vdev_id, inst_id, meas, &res);
+
+	return res.a0;
+}
+
 #endif /* __ASM_RSI_CMDS_H */
arch/arm64/include/asm/rsi_smc.h +16 -0
index f6aa647239c0..f051db54cdc3 100644
@@ -202,6 +202,22 @@ struct rsi_host_call {
 
 #define SMC_RSI_RDEV_GET_INTERFACE_REPORT	SMC_RSI_FID(0x1a6)
 
+#define RSI_DEV_MEASURE_ALL		BIT(0)
+#define RSI_DEV_MEASURE_SIGNED		BIT(1)
+#define RSI_DEV_MEASURE_RAW		BIT(2)
+
+struct rsi_device_measurements_params {
+	union {
+		struct {
+			u64 flags;
+			u8 indices[32];
+			u8 nounce[32];
+		};
+		u8 padding[0x100];
+	};
+};
+
+#define SMC_RSI_RDEV_GET_MEASUREMENTS		SMC_RSI_FID(0x1a7)
 #define SMC_RSI_RDEV_LOCK			SMC_RSI_FID(0x1a9)
 #define SMC_RSI_RDEV_START			SMC_RSI_FID(0x1aa)
 #define SMC_RSI_RDEV_STOP			SMC_RSI_FID(0x1ab)
drivers/virt/coco/arm-cca-guest/rsi-da.c +39 -0
index 64034d220e02..6222b10964ee 100644
@@ -166,10 +166,31 @@ static long rhi_get_report(int vdev_id, int da_object_type, void **report, int *
 	return ret;
 }
 
+static inline unsigned long
+rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id,
+			  unsigned long inst_id, phys_addr_t meas)
+{
+	unsigned long ret;
+
+	ret = __rsi_rdev_get_measurements(vdev_id, inst_id, meas);
+	if (ret != RSI_SUCCESS)
+		return ret;
+
+	do {
+		ret = rsi_rdev_continue(vdev_id, inst_id);
+	} while (ret == RSI_INCOMPLETE);
+	if (ret != RSI_SUCCESS) {
+		pci_err(pdev, "failed to communicate with the device (%lu)\n", ret);
+		return ret;
+	}
+	return RSI_SUCCESS;
+}
+
 int rsi_device_lock(struct pci_dev *pdev)
 {
 	unsigned long ret;
 	unsigned long tdisp_version;
+	struct rsi_device_measurements_params *rsi_dev_meas;
 	struct cca_guest_dsc *dsm = to_cca_guest_dsc(pdev);
 	int vdev_id = (pci_domain_nr(pdev->bus) << 16) |
 		PCI_DEVID(pdev->bus->number, pdev->devfn);
@@ -198,6 +219,17 @@ int rsi_device_lock(struct pci_dev *pdev)
 		return -EOPNOTSUPP;
 	}
 
+	rsi_dev_meas = (struct rsi_device_measurements_params *)__get_free_page(GFP_KERNEL);
+	rsi_dev_meas->flags = RSI_DEV_MEASURE_ALL;
+	ret = rsi_rdev_get_measurements(pdev, vdev_id, dsm->instance_id,
+					virt_to_phys(rsi_dev_meas));
+
+	free_page((unsigned long)rsi_dev_meas);
+	if (ret != RSI_SUCCESS) {
+		pci_err(pdev, "failed to get device measurement (%lu)\n", ret);
+		return -EIO;
+	}
+
 	/* Now make a host call to copy the interface report to guest. */
 	ret = rhi_get_report(vdev_id, RHI_DA_OBJECT_INTERFACE_REPORT,
 			     &dsm->interface_report, &dsm->interface_report_size);
@@ -213,6 +245,13 @@ int rsi_device_lock(struct pci_dev *pdev)
 		return -EIO;
 	}
 
+	ret = rhi_get_report(vdev_id, RHI_DA_OBJECT_MEASUREMENT,
+			     &dsm->measurements, &dsm->measurements_size);
+	if (ret) {
+		pci_err(pdev, "failed to get device certificate from the host (%lu)\n", ret);
+		return -EIO;
+	}
+
 	return ret;
 }
 static inline unsigned long rsi_rdev_start(struct pci_dev *pdev,
drivers/virt/coco/arm-cca-guest/rsi-da.h +2 -0
index 71ee1edb832e..f26156d9be81 100644
@@ -40,6 +40,8 @@ struct cca_guest_dsc {
 	int interface_report_size;
 	void *certificate;
 	int certificate_size;
+	void *measurements;
+	int measurements_size;
 };
 
 static inline struct cca_guest_dsc *to_cca_guest_dsc(struct pci_dev *pdev)
Signature · 1 line
2.43.0
#167 lore raw

Re: [RFC PATCH v1 37/38] coco: guest: arm64: Add support for fetching device measurements

Quoted 3 lines · On Mon, 28 Jul 2025 19:22:14 +0530 "Aneesh Kumar K.V (Arm)"…

Fetch device measurements using RSI_RDEV_GET_MEASUREMENTS.

One completely trivial comment.

J

Quoted 7 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +0 -0
index 64034d220e02..6222b10964ee 100644
@@ -213,6 +245,13 @@ int rsi_device_lock(struct pci_dev *pdev)
 		return -EIO;

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 255

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +7 -0
 	}
 
+	ret = rhi_get_report(vdev_id, RHI_DA_OBJECT_MEASUREMENT,
+			     &dsm->measurements, &dsm->measurements_size);
+	if (ret) {
+		pci_err(pdev, "failed to get device certificate from the host (%lu)\n", ret);
+		return -EIO;
+	}
+
 	return ret;

return 0; Always good to make it explicit when it can't take any other values. Looks like that belong sin an earlier patch though based on this snippet.

Quoted 1 line
drivers/virt/coco/arm-cca-guest/rsi-da.c +0 -0
 }
#168 lore raw

Re: [RFC PATCH v1 37/38] coco: guest: arm64: Add support for fetching device measurements

In reply to Aneesh Kumar K.V (Arm)

arch/arm64/include/asm/rsi_smc.h struct rsi_host_call { · line 214

Quoted 10 lines · Aneesh Kumar K.V (Arm)

Fetch device measurements using RSI_RDEV_GET_MEASUREMENTS.

+++ b/arch/arm64/include/asm/rsi_smc.h

arch/arm64/include/asm/rsi_smc.h +6 -0
+struct rsi_device_measurements_params {
+	union {
+		struct {
+			u64 flags;
+			u8 indices[32];
+			u8 nounce[32];

s/nounce/nonce/ ?

author maintainer
#169 lore raw

RFC PATCH 38/38 coco: guest: arm64: Add support for fetching device info

In reply to Aneesh Kumar K.V (Arm)

RSI_RDEV_GET_INFO returns different digest hash values, which can be compared with host cached values to ensure the host didn't tamper with the cached data.

---
 arch/arm64/include/asm/rsi_cmds.h        |  12 +++
 arch/arm64/include/asm/rsi_smc.h         |  24 +++++
 drivers/virt/coco/arm-cca-guest/Kconfig  |   2 +
 drivers/virt/coco/arm-cca-guest/rsi-da.c | 128 +++++++++++++++++++++++
 drivers/virt/coco/arm-cca-guest/rsi-da.h |  13 +++
 5 files changed, 179 insertions(+)
arch/arm64/include/asm/rsi_cmds.h +12 -0
index 42b998f44a0e..6b90a6cdd7fb 100644
@@ -276,4 +276,16 @@ static inline unsigned long __rsi_rdev_get_measurements(unsigned long vdev_id,
 	return res.a0;
 }
 
+static inline unsigned long rsi_rdev_get_info(unsigned long vdev_id,
+					      unsigned long inst_id,
+					      unsigned long digest_phys)
+{
+	struct arm_smccc_res res;
+
+	arm_smccc_1_1_invoke(SMC_RSI_RDEV_GET_INFO,
+			     vdev_id, inst_id, digest_phys, &res);
+
+	return res.a0;
+}
+
 #endif /* __ASM_RSI_CMDS_H */
arch/arm64/include/asm/rsi_smc.h +24 -0
index f051db54cdc3..e2b51cf58bd4 100644
@@ -125,6 +125,9 @@
 
 #ifndef __ASSEMBLY__
 
+#define RSI_HASH_SHA_256 0
+#define RSI_HASH_SHA_512 1
+
 struct realm_config {
 	union {
 		struct {
@@ -200,6 +203,27 @@ struct rsi_host_call {
 #define SMC_RSI_RDEV_GET_INSTANCE_ID		SMC_RSI_FID(0x19c)
 #define SMC_RSI_RDEV_CONTINUE			SMC_RSI_FID(0x1a4)
 
+struct rsi_device_info {
+	union {
+		struct {
+			u64 flags;
+			u64 attest_type;
+			u64 cert_id;
+			u8 hash_algo;
+		};
+		u8 padding[0x40];
+	};
+	union { /* 0x40  */
+		struct {
+			u8 cert_digest[0x40];
+			u8 meas_digest[0x40];
+			u8 report_digest[0x40];
+		};
+		u8 padding2[0x200 - 0x40];
+	};
+};
+
+#define SMC_RSI_RDEV_GET_INFO			SMC_RSI_FID(0x1a5)
 #define SMC_RSI_RDEV_GET_INTERFACE_REPORT	SMC_RSI_FID(0x1a6)
 
 #define RSI_DEV_MEASURE_ALL		BIT(0)
drivers/virt/coco/arm-cca-guest/Kconfig +2 -0
index 410d9c3fb2b3..6fc86c1f3900 100644
@@ -5,6 +5,8 @@ config ARM_CCA_GUEST
 	tristate "Arm CCA Guest driver"
 	depends on ARM64
 	depends on PCI_TSM
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
 	select TSM_REPORTS
 	select TSM
 	help
drivers/virt/coco/arm-cca-guest/rsi-da.c +128 -0

153 lines not on this page · raw message

drivers/virt/coco/arm-cca-guest/rsi-da.h +13 -0
index f26156d9be81..e8953b8e85a3 100644
@@ -10,6 +10,7 @@
 #include <linux/pci-tsm.h>
 #include <asm/rsi_smc.h>
 #include <asm/rhi.h>
+#include <crypto/sha2.h>
 
 struct pci_tdisp_device_interface_report {
 	u16 interface_info;
@@ -33,6 +34,17 @@ struct pci_tdisp_mmio_range {
 #define TSM_INTF_REPORT_MMIO_RESERVED		GENMASK(15, 4)
 #define TSM_INTF_REPORT_MMIO_RANGE_ID		GENMASK(31, 16)
 
+struct dsm_device_info {
+	u64 flags;
+	u64 attest_type;
+	u64 cert_id;
+	u64 hash_algo;
+	u8 cert_digest[SHA512_DIGEST_SIZE];
+	u8 meas_digest[SHA512_DIGEST_SIZE];
+	u8 report_digest[SHA512_DIGEST_SIZE];
+};
+
+
 struct cca_guest_dsc {
 	struct pci_tsm_pf0 pci;
 	unsigned long instance_id;
@@ -42,6 +54,7 @@ struct cca_guest_dsc {
 	int certificate_size;
 	void *measurements;
 	int measurements_size;
+	struct dsm_device_info dev_info;
 };
 
 static inline struct cca_guest_dsc *to_cca_guest_dsc(struct pci_dev *pdev)
Signature · 1 line
2.43.0
#170 lore raw

Re: [RFC PATCH v1 38/38] coco: guest: arm64: Add support for fetching device info

Quoted 5 lines · On Mon, 28 Jul 2025 19:22:15 +0530 "Aneesh Kumar K.V (Arm)"…

RSI_RDEV_GET_INFO returns different digest hash values, which can be compared with host cached values to ensure the host didn't tamper with the cached data.

Hi Aneesh

A few comments on this one

Jonathan

Quoted 41 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +26 -0
index 6222b10964ee..a1bb225adb4c 100644
@@ -6,6 +6,7 @@
 #include <linux/pci.h>
 #include <linux/mem_encrypt.h>
 #include <asm/rsi_cmds.h>
+#include <crypto/hash.h>
 
 #include "rsi-da.h"
 
@@ -186,11 +187,102 @@ rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id,
 	return RSI_SUCCESS;
 }
 
+static int verify_digests(struct cca_guest_dsc *dsm)
+{
+	int i;
+	int ret;
+	u8 digest[SHA512_DIGEST_SIZE];
+	int sdesc_size;
+	size_t digest_size;
+	char *hash_alg_name;
+	struct shash_desc *shash;
+	struct crypto_shash *alg;
+	struct pci_dev *pdev = dsm->pci.tsm.pdev;
+	struct {
+		uint8_t *report;
+		size_t size;
+		uint8_t *digest;
+	} reports[] = {
+		{
+			dsm->interface_report,
+			dsm->interface_report_size,
+			dsm->dev_info.report_digest
+		},
+		{
+			dsm->certificate,
+			dsm->certificate_size,
+			dsm->dev_info.cert_digest

drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 224

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+		},
+		{
+			dsm->measurements,
+			dsm->measurements_size,
+			dsm->dev_info.meas_digest
+		}
+	};
+
+
+	if (dsm->dev_info.hash_algo == RSI_HASH_SHA_256) {

This to me smells like a place that will need switch sooner or later, maybe just do it now.

drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 232

Quoted 8 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +8 -0
+		hash_alg_name = "sha256";
+		digest_size = SHA256_DIGEST_SIZE;
+	} else if (dsm->dev_info.hash_algo == RSI_HASH_SHA_512) {
+		hash_alg_name = "sha512";
+		digest_size = SHA512_DIGEST_SIZE;
+	} else {
+		pci_err(pdev, "unknown realm hash algorithm!\n");
+		ret = -EINVAL;

return -EINVAL;

drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 236

Quoted 4 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +4 -0
+		goto err_out;
+	}
+
+	alg = crypto_alloc_shash(hash_alg_name, 0, 0);

As below - I'd spin a DEFINE_FREE() for this to simplify error paths in here and remove the labels that had me confused briefly.

drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 242

Quoted 6 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +6 -0
+	if (IS_ERR(alg)) {
+		pci_err(pdev, "cannot allocate %s\n", hash_alg_name);
+		return PTR_ERR(alg);
+	}
+
+	sdesc_size = sizeof(struct shash_desc) + crypto_shash_descsize(alg);

Not common in crypto so perhaps just leave this as you have it.

	sdesc_size = struct_size(struct shash_dec, __ctx, crypto_shash_desc_size(alg));

is more informative on what is going on here.

drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 253

Quoted 11 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +11 -0
+	shash = kzalloc(sdesc_size, GFP_KERNEL);
+	if (!shash) {
+		pci_err(pdev, "cannot allocate sdesc\n");
+		ret = -ENOMEM;
+		goto err_free_shash;
+	}
+	shash->tfm = alg;
+
+	for (i = 0; i < ARRAY_SIZE(reports); i++) {
+		ret = crypto_shash_digest(shash, reports[i].report,
+					  reports[i].size, digest);

To me a bit marginal on whether this loop and the structures above are beneficial over straight line code.

Quoted 6 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +6 -0
+		if (ret) {
+			pci_err(pdev, "failed to compute digest, %d\n", ret);
+			goto err_free_sdesc;
+		}
+
+		if (memcmp(reports[i].digest, digest, digest_size)) {

drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 269

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+			pci_err(pdev, "invalid digest\n");
+			ret = -EINVAL;
+			goto err_free_sdesc;
+		}
+	}
+
+	kfree(shash);
+	crypto_free_shash(alg);
+
+	pci_info(pdev, "Successfully verified the digests\n");

debug.

drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 272

Quoted 3 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +3 -0
+	return 0;
+
+err_free_sdesc:

I'd tweak these labels if you keep them. This isn't freeing the sdesc.

drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 273

Quoted 1 line
drivers/virt/coco/arm-cca-guest/rsi-da.c +1 -0
+	kfree(shash);

Looks perfect for some __free() magic dust.

drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 275

Quoted 2 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +2 -0
+err_free_shash:
+	crypto_free_shash(alg);

DEFINE_FREE() needed for this but looks pretty uncontroversial.

drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 276

Quoted 1 line
drivers/virt/coco/arm-cca-guest/rsi-da.c +1 -0
+err_out:

As below. I'd not do this.

Quoted 13 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +4 -0
+	return ret;
+}
+
 int rsi_device_lock(struct pci_dev *pdev)
 {
 	unsigned long ret;
 	unsigned long tdisp_version;
 	struct rsi_device_measurements_params *rsi_dev_meas;
+	struct rsi_device_info *dev_info;
 	struct cca_guest_dsc *dsm = to_cca_guest_dsc(pdev);
 	int vdev_id = (pci_domain_nr(pdev->bus) << 16) |
 		PCI_DEVID(pdev->bus->number, pdev->devfn);

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 348

Quoted 5 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +2 -0
@@ -252,8 +344,44 @@ int rsi_device_lock(struct pci_dev *pdev)
 		return -EIO;
 	}
 
+	/* RMM expects sizeof(dev_info) (512 bytes) aligned address */
+	dev_info = kmalloc(sizeof(*dev_info), GFP_KERNEL);

Use a __free(kfree) here (and direct returns on errors) given it's freed in all paths and we don't care if it is freed before or after verifying the digests.

I'm being slow today, but what in that enforces the alignment? I guess it's that the structure happens to be big enough that it happens naturally?

I'd allocate max(512, sizeof(*dev_info)) to make it explicitly the case.

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+	if (!dev_info) {
+		ret = -ENOMEM;
+		goto err_out;
+	}
+
+	ret = rsi_rdev_get_info(vdev_id, dsm->instance_id, virt_to_phys(dev_info));
+	if (ret != RSI_SUCCESS) {
+		pci_err(pdev, "failed to get device digests (%lu)\n", ret);
+		ret = -EIO;
+		kfree(dev_info);

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 368

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+		goto err_out;
+	}
+
+	dsm->dev_info.attest_type   = dev_info->attest_type;
+	dsm->dev_info.cert_id       = dev_info->cert_id;
+	dsm->dev_info.hash_algo     = dev_info->hash_algo;
+	memcpy(dsm->dev_info.cert_digest, dev_info->cert_digest, SHA512_DIGEST_SIZE);
+	memcpy(dsm->dev_info.meas_digest, dev_info->meas_digest, SHA512_DIGEST_SIZE);
+	memcpy(dsm->dev_info.report_digest, dev_info->report_digest, SHA512_DIGEST_SIZE);
+

Can't you memcpy the whole thing in one go?

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 381

Quoted 13 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +13 -0
+	kfree(dev_info);
+	/*
+	 * Verify that the digests of the provided reports match with the
+	 * digests from RMM
+	 */
+	ret = verify_digests(dsm);
+	if (ret) {
+		pci_err(pdev, "device digest validation failed (%ld)\n", ret);
+		return ret;
+	}
+
+	return 0;
+err_out:

I'll always grumble about these. To me it always makes the code less readable. Some others disagree though ;(

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 384

Quoted 3 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +1 -0
 	return ret;
 }
+

Looks like this should have been in an earlier patch.

Quoted 19 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +0 -0
 static inline unsigned long rsi_rdev_start(struct pci_dev *pdev,
 					   unsigned long vdev_id, unsigned long inst_id)
 {
drivers/virt/coco/arm-cca-guest/rsi-da.h +1 -0
index f26156d9be81..e8953b8e85a3 100644
@@ -10,6 +10,7 @@
 #include <linux/pci-tsm.h>
 #include <asm/rsi_smc.h>
 #include <asm/rhi.h>
+#include <crypto/sha2.h>
 
 struct pci_tdisp_device_interface_report {
 	u16 interface_info;
@@ -33,6 +34,17 @@ struct pci_tdisp_mmio_range {
 #define TSM_INTF_REPORT_MMIO_RESERVED		GENMASK(15, 4)
 #define TSM_INTF_REPORT_MMIO_RANGE_ID		GENMASK(31, 16)
 

drivers/virt/coco/arm-cca-guest/rsi-da.h struct pci_tdisp_mmio_range { · line 46

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.h +10 -0
+struct dsm_device_info {
+	u64 flags;
+	u64 attest_type;
+	u64 cert_id;
+	u64 hash_algo;
+	u8 cert_digest[SHA512_DIGEST_SIZE];
+	u8 meas_digest[SHA512_DIGEST_SIZE];
+	u8 report_digest[SHA512_DIGEST_SIZE];
+};
+

One probably enough.

Quoted 12 lines
drivers/virt/coco/arm-cca-guest/rsi-da.h +2 -0
+
 struct cca_guest_dsc {
 	struct pci_tsm_pf0 pci;
 	unsigned long instance_id;
@@ -42,6 +54,7 @@ struct cca_guest_dsc {
 	int certificate_size;
 	void *measurements;
 	int measurements_size;
+	struct dsm_device_info dev_info;
 };
 
 static inline struct cca_guest_dsc *to_cca_guest_dsc(struct pci_dev *pdev)
author maintainer
#171 lore raw

Re: [RFC PATCH v1 38/38] coco: guest: arm64: Add support for fetching device info

Quoted 3 lines · Jonathan Cameron

On Mon, 28 Jul 2025 19:22:15 +0530
"Aneesh Kumar K.V (Arm)" <[email protected]> wrote:

...

Quoted 22 lines
Quoted 13 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +4 -0
+	return ret;
+}
+
 int rsi_device_lock(struct pci_dev *pdev)
 {
 	unsigned long ret;
 	unsigned long tdisp_version;
 	struct rsi_device_measurements_params *rsi_dev_meas;
+	struct rsi_device_info *dev_info;
 	struct cca_guest_dsc *dsm = to_cca_guest_dsc(pdev);
 	int vdev_id = (pci_domain_nr(pdev->bus) << 16) |
 		PCI_DEVID(pdev->bus->number, pdev->devfn);

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 348

Quoted 5 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +2 -0
@@ -252,8 +344,44 @@ int rsi_device_lock(struct pci_dev *pdev)
 		return -EIO;
 	}
 
+	/* RMM expects sizeof(dev_info) (512 bytes) aligned address */
+	dev_info = kmalloc(sizeof(*dev_info), GFP_KERNEL);

Use a __free(kfree) here (and direct returns on errors) given it's freed in all paths and we don't care if it is freed before or after verifying the digests.

Quoted 5 lines

I'm being slow today, but what in that enforces the alignment? I guess it's that the structure happens to be big enough that it happens naturally?

I'd allocate max(512, sizeof(*dev_info)) to make it explicitly the case.

yes, struct rsi_device_info is 512 bytes.

Quoted 15 lines
Quoted 15 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +14 -0
+	if (!dev_info) {
+		ret = -ENOMEM;
+		goto err_out;
+	}
+
+	ret = rsi_rdev_get_info(vdev_id, dsm->instance_id, virt_to_phys(dev_info));
+	if (ret != RSI_SUCCESS) {
+		pci_err(pdev, "failed to get device digests (%lu)\n", ret);
+		ret = -EIO;
+		kfree(dev_info);
+		goto err_out;
+	}
+
+	dsm->dev_info.attest_type   = dev_info->attest_type;
Quoted 9 lines

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 368

Quoted 6 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +6 -0
+	dsm->dev_info.cert_id       = dev_info->cert_id;
+	dsm->dev_info.hash_algo     = dev_info->hash_algo;
+	memcpy(dsm->dev_info.cert_digest, dev_info->cert_digest, SHA512_DIGEST_SIZE);
+	memcpy(dsm->dev_info.meas_digest, dev_info->meas_digest, SHA512_DIGEST_SIZE);
+	memcpy(dsm->dev_info.report_digest, dev_info->report_digest, SHA512_DIGEST_SIZE);
+

Can't you memcpy the whole thing in one go?

yes. But won't that be confusing? Is there a difference? Also struct dsm_device_info is not same as struct rsi_device_info. We don't need to keep all that padding in dsm_device_info.

Quoted 65 lines
Quoted 4 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +3 -0
+	kfree(dev_info);
+	/*
+	 * Verify that the digests of the provided reports match with the

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 381

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+	 * digests from RMM
+	 */
+	ret = verify_digests(dsm);
+	if (ret) {
+		pci_err(pdev, "device digest validation failed (%ld)\n", ret);
+		return ret;
+	}
+
+	return 0;
+err_out:

I'll always grumble about these. To me it always makes the code less readable. Some others disagree though ;(

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 384

Quoted 3 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +1 -0
 	return ret;
 }
+

Looks like this should have been in an earlier patch.

Quoted 19 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +0 -0
 static inline unsigned long rsi_rdev_start(struct pci_dev *pdev,
 					   unsigned long vdev_id, unsigned long inst_id)
 {
drivers/virt/coco/arm-cca-guest/rsi-da.h +1 -0
index f26156d9be81..e8953b8e85a3 100644
@@ -10,6 +10,7 @@
 #include <linux/pci-tsm.h>
 #include <asm/rsi_smc.h>
 #include <asm/rhi.h>
+#include <crypto/sha2.h>
 
 struct pci_tdisp_device_interface_report {
 	u16 interface_info;
@@ -33,6 +34,17 @@ struct pci_tdisp_mmio_range {
 #define TSM_INTF_REPORT_MMIO_RESERVED		GENMASK(15, 4)
 #define TSM_INTF_REPORT_MMIO_RANGE_ID		GENMASK(31, 16)
 

drivers/virt/coco/arm-cca-guest/rsi-da.h struct pci_tdisp_mmio_range { · line 46

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.h +10 -0
+struct dsm_device_info {
+	u64 flags;
+	u64 attest_type;
+	u64 cert_id;
+	u64 hash_algo;
+	u8 cert_digest[SHA512_DIGEST_SIZE];
+	u8 meas_digest[SHA512_DIGEST_SIZE];
+	u8 report_digest[SHA512_DIGEST_SIZE];
+};
+

One probably enough.

Quoted 11 lines
drivers/virt/coco/arm-cca-guest/rsi-da.h +2 -0
+
 struct cca_guest_dsc {
 	struct pci_tsm_pf0 pci;
 	unsigned long instance_id;
@@ -42,6 +54,7 @@ struct cca_guest_dsc {
 	int certificate_size;
 	void *measurements;
 	int measurements_size;
+	struct dsm_device_info dev_info;
 };
 
Quoted 1 line

drivers/virt/coco/arm-cca-guest/rsi-da.h struct cca_guest_dsc { · line 60

Quoted 1 line
drivers/virt/coco/arm-cca-guest/rsi-da.h +0 -0
 static inline struct cca_guest_dsc *to_cca_guest_dsc(struct pci_dev *pdev)

-aneesh

#172 lore raw

Re: [RFC PATCH v1 38/38] coco: guest: arm64: Add support for fetching device info

Quoted 26 lines
Quoted 16 lines
Quoted 16 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +14 -0
+	if (!dev_info) {
+		ret = -ENOMEM;
+		goto err_out;
+	}
+
+	ret = rsi_rdev_get_info(vdev_id, dsm->instance_id, virt_to_phys(dev_info));
+	if (ret != RSI_SUCCESS) {
+		pci_err(pdev, "failed to get device digests (%lu)\n", ret);
+		ret = -EIO;
+		kfree(dev_info);
+		goto err_out;
+	}
+
+	dsm->dev_info.attest_type   = dev_info->attest_type;
Quoted 9 lines

drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 368

Quoted 6 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +6 -0
+	dsm->dev_info.cert_id       = dev_info->cert_id;
+	dsm->dev_info.hash_algo     = dev_info->hash_algo;
+	memcpy(dsm->dev_info.cert_digest, dev_info->cert_digest, SHA512_DIGEST_SIZE);
+	memcpy(dsm->dev_info.meas_digest, dev_info->meas_digest, SHA512_DIGEST_SIZE);
+	memcpy(dsm->dev_info.report_digest, dev_info->report_digest, SHA512_DIGEST_SIZE);
+  

Can't you memcpy the whole thing in one go?

Quoted 3 lines

yes. But won't that be confusing? Is there a difference? Also struct dsm_device_info is not same as struct rsi_device_info. We don't need to keep all that padding in dsm_device_info.

Ah. I misread and thought they were the same structure. No problem copying only relevant fields then!

Jonathan

#173 lore raw

Re: [RFC PATCH v1 38/38] coco: guest: arm64: Add support for fetching device info

In reply to Aneesh Kumar K.V (Arm)

drivers/virt/coco/arm-cca-guest/Kconfig config ARM_CCA_GUEST · line 11

Quoted 8 lines · Aneesh Kumar K.V (Arm)
drivers/virt/coco/arm-cca-guest/Kconfig +2 -0
@@ -5,6 +5,8 @@ config ARM_CCA_GUEST
 	tristate "Arm CCA Guest driver"
 	depends on ARM64
 	depends on PCI_TSM
+	select CRYPTO_SHA256
+	select CRYPTO_SHA512
 	select TSM_REPORTS
 	select TSM

CRYPTO_LIB_SHA256 and CRYPTO_LIB_SHA512

Quoted 24 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +24 -0
+	if (dsm->dev_info.hash_algo == RSI_HASH_SHA_256) {
+		hash_alg_name = "sha256";
+		digest_size = SHA256_DIGEST_SIZE;
+	} else if (dsm->dev_info.hash_algo == RSI_HASH_SHA_512) {
+		hash_alg_name = "sha512";
+		digest_size = SHA512_DIGEST_SIZE;
+	} else {
+		pci_err(pdev, "unknown realm hash algorithm!\n");
+		ret = -EINVAL;
+		goto err_out;
+	}
+
+	alg = crypto_alloc_shash(hash_alg_name, 0, 0);
+	if (IS_ERR(alg)) {
+		pci_err(pdev, "cannot allocate %s\n", hash_alg_name);
+		return PTR_ERR(alg);
+	}
+
+	sdesc_size = sizeof(struct shash_desc) + crypto_shash_descsize(alg);
+	shash = kzalloc(sdesc_size, GFP_KERNEL);
+	if (!shash) {
+		pci_err(pdev, "cannot allocate sdesc\n");
+		ret = -ENOMEM;
+		goto err_free_shash;

drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 257

Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10 -0
+	}
+	shash->tfm = alg;
+
+	for (i = 0; i < ARRAY_SIZE(reports); i++) {
+		ret = crypto_shash_digest(shash, reports[i].report,
+					  reports[i].size, digest);
+		if (ret) {
+			pci_err(pdev, "failed to compute digest, %d\n", ret);
+			goto err_free_sdesc;
+		}

Use sha256() and sha512().

- Eric

#174 lore raw
In reply to Aneesh Kumar K.V (Arm)
Quoted 3 lines · Aneesh Kumar K.V (Arm)

This patch series implements support for Device Assignment in the ARM CCA architecture. The code changes are based on Alp12 specification published here [1].

Robin and I were talking about CCA and DMA here:

https://lore.kernel.org/r/[email protected]

What do you think about pulling some of this out and trying to independently push a series getting the DMA API layers ready for device assignment?

I think there will be some discussion on these points, it would be good to get started.

Jason

#175 lore raw
In reply to Aneesh Kumar K.V (Arm)
Quoted 9 lines · Aneesh Kumar K.V (Arm)

This patch series implements support for Device Assignment in the ARM CCA architecture. The code changes are based on Alp12 specification published here [1].

The code builds on the TSM framework patches posted at [2]. We add extension to that framework so that TSM is now used in both the host and the guest.

A DA workflow can be summarized as below:

Quoted 8 lines

Host:
step 1.
echo ${DEVICE} > /sys/bus/pci/devices/${DEVICE}/driver/unbind echo vfio-pci > /sys/bus/pci/devices/${DEVICE}/driver_override echo ${DEVICE} > /sys/bus/pci/drivers_probe

step 2.
echo 1 > /sys/bus/pci/devices/$DEVICE/tsm/connect

Just for my own understanding... presumably there is no ordering constraint for ARM CCA between step1 and step2, right? I.e. The connect state is independent of the bind state.

In the v4 PCI/TSM scheme the connect command is now:

echo $tsm_dev > /sys/bus/pci/devices/$DEVICE/tsm/connect

Quoted 1 line

Now in the guest we follow the below steps

I assume a signifcant amount of kvmtool magic happens here to get the TDI into a "bind capable" state, can you share that command?

I had been assuming that everyone was prototyping with QEMU. Not a problem per se, but the memory management for shared device assignment / bounce buffering has had a quite of bit of work on the QEMU side, so just curious about the difference in approach here. Like, does kvmtool support operating the device in shared mode with bounce buffering and page conversion (shared <=> private) support? In any event, happy to see mutiple simultaneous consumers of this new kernel infrastructure.

Quoted 5 lines

step 1:
echo ${DEVICE} > /sys/bus/pci/devices/${DEVICE}/driver/unbind

step 2: Move the device to TDISP LOCK state
echo 1 > /sys/bus/pci/devices/${DEVICE}/tsm/lock

Ok, so my stance has recently picked up some nuance here. As Jason mentions here:

http://lore.kernel.org/[email protected]

"However it works, it should be done before the driver is probed and remain stable for the duration of the driver attachment. From the iommu side the correct iommu domain, on the correct IOMMU instance to handle the expected traffic should be setup as the DMA API's iommu domain."

I agree with that up until the point where the implication is userspace control of the UNLOCKED->LOCKED transition. That transition requires enabling bus-mastering (BME), configuring the device into an expected state, and *then* locking the device. That means userspace is blindly hoping that the device is in a state where it will remain quiet on the bus between BME and LOCKED, and that the previous unbind left the device in a state where it is prepared to be locked again.

The BME concern may be overblown given major PCI drivers blindly set BME without validating the device is in a quiesced state, but the "device is prepped for locking" problem seems harder.

2 potential ways to solve this, but open to other ideas:

- Userspace only picks the iommu domain context for the device not the lock state. Something like:

  private > /sys/bus/pci/devices/${DEVICE}/tsm/domain

...where the default is "shared" and from that point the device can not issue DMA until a driver attaches. Driver controls UNLOCKED->LOCKED->RUN.

- Userspace is not involved in this transition and the dma mapping API is updated to allow a driver to switch the iommu domain at runtime, but only if the device has no outstanding mappings and the transition can only happen from ->probe() context. Driver controls joining secure-world-DMA and UNLOCKED->LOCKED->RUN.

Clearly the first option is less work in the kernel, but in both options the driver is in control of when BME is set relative to being ready for the LOCKED transition.

Quoted 2 lines

step 3: Moves the device to TDISP RUN state echo 1 > /sys/bus/pci/devices/${DEVICE}/tsm/accept

This has the same concern from me about userspace being in control of BME. It feels like a departure from typical expectations. At least in the case of a driver setting BME the driver's probe routine is going to get the device in order shortly and otherwise have error handlers at the ready to effect any needed recovery.

Userspace just leaves the device enabled indefinitely and hopes.

Now, the nice thing about the scheme as proposed in this set is that userspace has all the time in the world between "lock" and "accept" to talk to a verifier.

With the driver in control there would need to be something like a usermodehelper to notify userspace that the device is in the locked state and to go ahead and run the attestation while the driver waits*.

* or driver could decide to not wait, especially useful for debug and development

Quoted 2 lines

step 4: Load the driver again.
echo ${DEVICE} > /sys/bus/pci/drivers_probe

TIL drivers_probe

Maybe want to recommend:

echo ${DEVICE} > /sys/bus/pci/drivers/${DRIVER}/bind

...to users just in case there are multiple drivers loaded for the device for the "shared" vs "private" case?

Quoted 2 lines

I'm currently working against TSM v3, as TSM v4 lacks the necessary callbacks—bind, unbind, and guest_req—required for guest interactions.

For staging purposes I wanted to put the "connect" flow to bed before moving on to the guest side.

Quoted 16 lines

The implementation also makes use of RHI interfaces that fall outside the current RHI specification [5]. Once the spec is finalized, the code will be aligned accordingly.

For now, I’ve retained validate_mmio and vdev_req exit handling within KVM. This will transition to a guest_req-based mechanism once the specification is updated.

At that point, all device assignment (DA)-specific VM exits will exit directly to the VMM, and will use the guest_req ioctl to handle exit reasons. As part of this change, the handlers realm_exit_vdev_req_handler, realm_exit_vdev_comm_handler, and realm_exit_dev_mem_map_handler will be removed.

Full patchset for the kernel and kvmtool can be found at [3] and [4]

Quoted 7 lines

[1] https://developer.arm.com/-/cdn-downloads/permalink/Architectures/Armv9/DEN0137_1.1-alp12.zip

[2] https://lore.kernel.org/all/[email protected]

[3] https://git.gitlab.arm.com/linux-arm/linux-cca.git cca/tdisp-upstream-post-v1
[4] https://git.gitlab.arm.com/linux-arm/kvmtool-cca.git cca/tdisp-upstream-post-v1
[5] https://developer.arm.com/documentation/den0148/latest/

Thanks for this and the help reviewing PCI/TSM so far! I want to get this into tsm.git#staging so we can start to make hard claims ("look at the shared tree!") of hardware vendor consensus.

#176 lore raw
Quoted 10 lines · On Thu, Jul 31, 2025 at 07:07:17PM -0700, [email protected] wrote:
Quoted 8 lines · Aneesh Kumar K.V (Arm)

Host:
step 1.
echo ${DEVICE} > /sys/bus/pci/devices/${DEVICE}/driver/unbind echo vfio-pci > /sys/bus/pci/devices/${DEVICE}/driver_override echo ${DEVICE} > /sys/bus/pci/drivers_probe

step 2.
echo 1 > /sys/bus/pci/devices/$DEVICE/tsm/connect

Quoted 7 lines

Just for my own understanding... presumably there is no ordering constraint for ARM CCA between step1 and step2, right? I.e. The connect state is independent of the bind state.

In the v4 PCI/TSM scheme the connect command is now:

echo $tsm_dev > /sys/bus/pci/devices/$DEVICE/tsm/connect

What does this do on the host? It seems to somehow prep it for VM assignment? Seems pretty strange this is here in sysfs and not part of creating the vPCI function in the VM through VFIO and iommufd?

Frankly, I'm nervous about making any uAPI whatsoever for the hypervisor side at this point. I don't think we have enough of the solution even in draft format. I'd really like your first merged TSM series to only have uAPI for the guest side where things are hopefully closer to complete..

Quoted 9 lines
Quoted 5 lines

step 1:
echo ${DEVICE} > /sys/bus/pci/devices/${DEVICE}/driver/unbind

step 2: Move the device to TDISP LOCK state
echo 1 > /sys/bus/pci/devices/${DEVICE}/tsm/lock

Ok, so my stance has recently picked up some nuance here. As Jason mentions here:

Quoted 7 lines

http://lore.kernel.org/[email protected]

"However it works, it should be done before the driver is probed and remain stable for the duration of the driver attachment. From the iommu side the correct iommu domain, on the correct IOMMU instance to handle the expected traffic should be setup as the DMA API's iommu domain."

I think it is not just the dma api, but also the MMIO registers may move location (form shared to protected IPA space for example). Meaning any attached driver is completely wrecked.

Quoted 3 lines

I agree with that up until the point where the implication is userspace control of the UNLOCKED->LOCKED transition. That transition requires enabling bus-mastering (BME),

Why? That's sad. BME should be controlled by the VM driver not the TSM, and it should be set only when a VM driver is probed to the RUN state device?

Quoted 4 lines

and *then* locking the device. That means userspace is blindly hoping that the device is in a state where it will remain quiet on the bus between BME and LOCKED, and that the previous unbind left the device in a state where it is prepared to be locked again.

Yes, but we broadly assume this already in Linux. Drivers assume their devices are quiet when they are bound the first time, we expect on unbinding a driver quiets the device before removing.

So broadly I think you can assume that a device with no driver is quiet regardless of BME.

Quoted 10 lines

2 potential ways to solve this, but open to other ideas:

- Userspace only picks the iommu domain context for the device not the lock state. Something like:

  private > /sys/bus/pci/devices/${DEVICE}/tsm/domain

...where the default is "shared" and from that point the device can not issue DMA until a driver attaches. Driver controls UNLOCKED->LOCKED->RUN.

What? Gross, no way can we let userspace control such intimate details of the kernel. The kernel must auto set based on what T=x mode the device driver binds into.

Quoted 5 lines

- Userspace is not involved in this transition and the dma mapping API is updated to allow a driver to switch the iommu domain at runtime, but only if the device has no outstanding mappings and the transition can only happen from ->probe() context. Driver controls joining secure-world-DMA and UNLOCKED->LOCKED->RUN.

I don't see why it is so complicated. The driver is unbound before it reaches T=1 so we expect the device to be quiet (bigger problems if not). When the PCI core reaches T=1 it tells the DMA API to reconfigure things for the unbound struct device. Then we bind a driver as normal.

Driver controls nothing. All existing T=0 drivers "just work" with no source changes in T=1 mode. DMA API magically hides the bounce buffering. Surely this should be the baseline target functionality from a Linux perspective?

So we should not have "driver controls" statements at all. Userspace prepares the PCI device, driver probes onto a T=1 environment and just works.

Quoted 5 lines
Quoted 2 lines

step 3: Moves the device to TDISP RUN state echo 1 > /sys/bus/pci/devices/${DEVICE}/tsm/accept

This has the same concern from me about userspace being in control of BME. It feels like a departure from typical expectations.

It is, it is architecturally broken for BME to be controlled by the TSM. BME is controlled by the guest OS driver only.

IMHO if this is a real worry (and I don't think it is) then the right answer is for physical BME to be set on during locking, but VIRTUAL BME is left off. Virtual BME is created by the hypervisor/tsm by telling the IOMMU to block DMA.

The Guest OS should not participate in this broken design, the hypervisor can set pBME automatically when the lock request comes in, and the quality of vBME emulation is left up to the implementation, but the implementation must provide at least a NOP vBME once locked.

Quoted 3 lines

Now, the nice thing about the scheme as proposed in this set is that userspace has all the time in the world between "lock" and "accept" to talk to a verifier.

Seems right to me. There should be NO trusted kernel driver bound until the verifier accepts the attestation. Anything else allows un accepted devices to attack the kernel drivers. Few kernel drivers today distrust their HW interfaces as hostile actors and security defend against them. Therefore we should be very reluctant to bind drivers to anything..

Arguably a CC secure kernel should have an allow list of audited secure drivers that can autoprobe and all other drivers must be approved by userspace in some way, either through T=1 and attestation or some customer-aware risk assumption.

From that principal the kernel should NOT auto probe drivers to T=0 devices that can be made T=1. Userspace should handle attaching HW to such devices, and userspace can sequence whatever is required, including the attestation and verifying.

Otherwise, if you say, have a TDISP capable mlx5 device and boot up the cVM in a comporomised host the host can probably completely hack your cVM by exploiting the mlx5 drivers's total trust in the HW interface while running in T=0 mode.

You must attest it and switch to T=1 before binding any driver if you care about mitigating this risk.

Quoted 3 lines

With the driver in control there would need to be something like a usermodehelper to notify userspace that the device is in the locked state and to go ahead and run the attestation while the driver waits*.

It doesn't make sense to require modification to all existing drivers in Linux! The starting point must have the core code do this sequence for every driver. Once that is working we can talk about if other flows are needed.

Quoted 11 lines
Quoted 2 lines

step 4: Load the driver again.
echo ${DEVICE} > /sys/bus/pci/drivers_probe

TIL drivers_probe

Maybe want to recommend:

echo ${DEVICE} > /sys/bus/pci/drivers/${DRIVER}/bind

...to users just in case there are multiple drivers loaded for the device for the "shared" vs "private" case?

Generic userspace will have a hard time to know what the driver names are..

The driver_probe option looks good to me as the default.

I'm not sure how generic code can handle "multiple drivers".. Most devices will be able to work just fine with T=0 mode with bounce buffers so we should generally not encourage people to make completely different drivers for T=0/T=1 mode.

I think what is needed is some way for userspace to trigger the "locking configuration" you mentioned, that may need a special driver, but ONLY if the userspace is sequencing the device to T=1 mode. Not sure how to make that generic, but I think so long as userspace is explicitly controlling driver binding we can punt on that solution to the userspace project :)

The real nastyness is RAS - what do you do when the device falls out of RUN, the kernel driver should pretty much explode. But lots of people would like the kernel driver to stay alive and somehow we FLR, re-attest and "resume" the kernel driver without allowing any T=0 risks. For instance you can keep your netdev and just see a lot of lost packets while the driver thrashes.

But I think we can start with the idea that such RAS failures have to reload the driver too and work on improvements. Realistically few drivers have the sort of RAS features to consume this anyhow and maybe we introduce some "enhanced" driver mode to opt-into down the road.

Jason

#177 lore raw
Quoted 15 lines · Jason Gunthorpe
Quoted 14 lines · On Thu, Jul 31, 2025 at 07:07:17PM -0700, [email protected] wrote:
Quoted 8 lines · Aneesh Kumar K.V (Arm)

Host:
step 1.
echo ${DEVICE} > /sys/bus/pci/devices/${DEVICE}/driver/unbind echo vfio-pci > /sys/bus/pci/devices/${DEVICE}/driver_override echo ${DEVICE} > /sys/bus/pci/drivers_probe

step 2.
echo 1 > /sys/bus/pci/devices/$DEVICE/tsm/connect

Just for my own understanding... presumably there is no ordering constraint for ARM CCA between step1 and step2, right? I.e. The connect state is independent of the bind state.

Quoted 7 lines
Quoted 3 lines

In the v4 PCI/TSM scheme the connect command is now:

echo $tsm_dev > /sys/bus/pci/devices/$DEVICE/tsm/connect

What does this do on the host? It seems to somehow prep it for VM assignment? Seems pretty strange this is here in sysfs and not part of creating the vPCI function in the VM through VFIO and iommufd?

vPCI is out of the picture at this phase.

On the host this establishes an SPDM session and sets up link encryption (IDE) with the physical device. Leave VMs out of the picture, this capability in isolation is a useful property. It addresses the similar threat model that Intel Total Memory Encryption (TME) or AMD Secure Memory Encryption (SME) go after, i.e. interposer on a physical link capturing data in flight.

With that established then one can go futher to do the full TDISP dance.

Quoted 5 lines

Frankly, I'm nervous about making any uAPI whatsoever for the hypervisor side at this point. I don't think we have enough of the solution even in draft format. I'd really like your first merged TSM series to only have uAPI for the guest side where things are hopefully closer to complete.

Aligned. I am not comfortable merging any of this until we have that end to end reliably stable for a kernel cycle or 2. The proposal is soak all the vendor solutions together in tsm.git#staging.

Now, if the guest side graduates out of that staging before the host side, I am ok with that.

Quoted 17 lines
Quoted 9 lines
Quoted 5 lines

step 1:
echo ${DEVICE} > /sys/bus/pci/devices/${DEVICE}/driver/unbind

step 2: Move the device to TDISP LOCK state
echo 1 > /sys/bus/pci/devices/${DEVICE}/tsm/lock

Ok, so my stance has recently picked up some nuance here. As Jason mentions here:

Quoted 7 lines

http://lore.kernel.org/[email protected]

"However it works, it should be done before the driver is probed and remain stable for the duration of the driver attachment. From the iommu side the correct iommu domain, on the correct IOMMU instance to handle the expected traffic should be setup as the DMA API's iommu domain."

Quoted 3 lines

I think it is not just the dma api, but also the MMIO registers may move location (form shared to protected IPA space for example). Meaning any attached driver is completely wrecked.

True.

Quoted 7 lines
Quoted 3 lines

I agree with that up until the point where the implication is userspace control of the UNLOCKED->LOCKED transition. That transition requires enabling bus-mastering (BME),

Why? That's sad. BME should be controlled by the VM driver not the TSM, and it should be set only when a VM driver is probed to the RUN state device?

To me it is an unfortunate PCI specification wrinkle that writing to the command register drops the device from RUN to ERROR. So you can LOCK without setting BME, but then no DMA.

Quoted 19 lines
Quoted 4 lines

and *then* locking the device. That means userspace is blindly hoping that the device is in a state where it will remain quiet on the bus between BME and LOCKED, and that the previous unbind left the device in a state where it is prepared to be locked again.

Yes, but we broadly assume this already in Linux. Drivers assume their devices are quiet when they are bound the first time, we expect on unbinding a driver quiets the device before removing.

So broadly I think you can assume that a device with no driver is quiet regardless of BME.

Quoted 7 lines

2 potential ways to solve this, but open to other ideas:

- Userspace only picks the iommu domain context for the device not the lock state. Something like:

  private > /sys/bus/pci/devices/${DEVICE}/tsm/domain
Quoted 7 lines
Quoted 3 lines

...where the default is "shared" and from that point the device can not issue DMA until a driver attaches. Driver controls UNLOCKED->LOCKED->RUN.

What? Gross, no way can we let userspace control such intimate details of the kernel. The kernel must auto set based on what T=x mode the device driver binds into.

Flummoxed. Any way this gets sliced, userspace is asking for "private world attach" because it alone knows that this device is acceptable, and devices need to arrive in "shared world attach" mode.

Quoted 12 lines
Quoted 5 lines

- Userspace is not involved in this transition and the dma mapping API is updated to allow a driver to switch the iommu domain at runtime, but only if the device has no outstanding mappings and the transition can only happen from ->probe() context. Driver controls joining secure-world-DMA and UNLOCKED->LOCKED->RUN.

I don't see why it is so complicated. The driver is unbound before it reaches T=1 so we expect the device to be quiet (bigger problems if not). When the PCI core reaches T=1 it tells the DMA API to reconfigure things for the unbound struct device. Then we bind a driver as normal.

Quoted 4 lines

Driver controls nothing. All existing T=0 drivers "just work" with no source changes in T=1 mode. DMA API magically hides the bounce buffering. Surely this should be the baseline target functionality from a Linux perspective?

I started this project with "all existing T=0 drivers 'just work'" as a goal and a virtue. I have been begrudgingly pulled away from it from the slow drip of complexity it appears to push into the PCI core.

Now, I suspect the number of devices that are willing to spend gates and firmware on TDISP capabilities in the near term is small. The "just works" case is saved for either an L1 VMM to hide all this from an L2 guest, or a simplified TDISP specification that actually allows an OS PCI core to handle these details in a standard way.

Quoted 3 lines

So we should not have "driver controls" statements at all. Userspace prepares the PCI device, driver probes onto a T=1 environment and just works.

The concern is neither userspace nor the PCI core have everything it needs to get the device to T=1. PCI core knows that the device is T=1 capable, but does not know how to preconfigure the device-specific lock state, needs to wait for attestation. Userpace knows how to attest/verify the device but really has no business running the device outside of binding a driver, and can not rely on the PCI core to have prepped the device's device-specific lock state.

Userspace might be able to bind a new driver that leaves the device in a lockable state on unbind, but that is not "just works" that is, "introduce a new concept of skinny TDISP setup drivers that leave devices in LOCKED state on driver unbind, so that userspace can do the work to verify the device and move it to RUN before loading the main driver that expects the device arrives already running. Also, that main driver needs to be careful not to trigger typically benign actions like touch the command register to trip the device into ERROR state, or any device-specific actions that trip ERROR state but would otherwise be benign outside of TDISP."

If locking the device was just a toggle it would be possible. As far as I can see it is a "prep+toggle" where "prep" needs a driver.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

step 3: Moves the device to TDISP RUN state echo 1 > /sys/bus/pci/devices/${DEVICE}/tsm/accept

This has the same concern from me about userspace being in control of BME. It feels like a departure from typical expectations.

It is, it is architecturally broken for BME to be controlled by the TSM. BME is controlled by the guest OS driver only.

Agree. That "accept" attribute does not belong with TSM. That is where Aneesh has it in this RFC. "Accept" as an action is the combination of device entered the LOCKED state in a configuration the verifier is willing to accept and the mechanics of triggering the LOCKED->RUN transition.

Quoted 9 lines

IMHO if this is a real worry (and I don't think it is) then the right answer is for physical BME to be set on during locking, but VIRTUAL BME is left off. Virtual BME is created by the hypervisor/tsm by telling the IOMMU to block DMA.

The Guest OS should not participate in this broken design, the hypervisor can set pBME automatically when the lock request comes in, and the quality of vBME emulation is left up to the implementation, but the implementation must provide at least a NOP vBME once locked.

I can let go of the "BME without driver" worry, but that does nothing to solve the "device specific configuration required before lock" problem.

Quoted 11 lines
Quoted 3 lines

Now, the nice thing about the scheme as proposed in this set is that userspace has all the time in the world between "lock" and "accept" to talk to a verifier.

Seems right to me. There should be NO trusted kernel driver bound until the verifier accepts the attestation. Anything else allows un accepted devices to attack the kernel drivers. Few kernel drivers today distrust their HW interfaces as hostile actors and security defend against them. Therefore we should be very reluctant to bind drivers to anything.

Quoted 4 lines

Arguably a CC secure kernel should have an allow list of audited secure drivers that can autoprobe and all other drivers must be approved by userspace in some way, either through T=1 and attestation or some customer-aware risk assumption.

Yes, today, where nothing is T=1 capable for an L1 guest*, the onus is 100% on the distribution, not the kernel. I.e. trim kernel config and set modprobe policy to prevent unwanted drivers.

* For L2 there are proposals like this, where if you already trust your paravisor also pre-trust all the devices it tells you to trust.
[1]: http://lore.kernel.org/[email protected]

Quoted 4 lines

From that principal the kernel should NOT auto probe drivers to T=0 devices that can be made T=1. Userspace should handle attaching HW to such devices, and userspace can sequence whatever is required, including the attestation and verifying.

Agree, for PCI it would be simple to set a no-auto-probe policy for T=1 capable devices.

Quoted 7 lines

Otherwise, if you say, have a TDISP capable mlx5 device and boot up the cVM in a comporomised host the host can probably completely hack your cVM by exploiting the mlx5 drivers's total trust in the HW interface while running in T=0 mode.

You must attest it and switch to T=1 before binding any driver if you care about mitigating this risk.

Yes, userspace must have a chance to say "no" before a driver attempts to launch DMA to private memory after secrets have been deployed to the TVM.

Quoted 6 lines
Quoted 3 lines

With the driver in control there would need to be something like a usermodehelper to notify userspace that the device is in the locked state and to go ahead and run the attestation while the driver waits*.

It doesn't make sense to require modification to all existing drivers in Linux!

I do not want to burden the PCI core with TDISP compatibility hacks and workarounds if it turns out only a small handful of devices ever deploy a first generation TDISP Device Security Manager (DSM). L1 aiding L2, or TDISP simplicity improvements to allow the PCI core to handle this in a non-broken way, are what I expect if secure device assignment takes off.

Quoted 3 lines

The starting point must have the core code do this sequence for every driver. Once that is working we can talk about if other flows are needed.

Do you agree that "device-specific-prep+lock" is the problem to solve?

Quoted 29 lines
Quoted 11 lines
Quoted 2 lines

step 4: Load the driver again.
echo ${DEVICE} > /sys/bus/pci/drivers_probe

TIL drivers_probe

Maybe want to recommend:

echo ${DEVICE} > /sys/bus/pci/drivers/${DRIVER}/bind

...to users just in case there are multiple drivers loaded for the device for the "shared" vs "private" case?

Generic userspace will have a hard time to know what the driver names are..

The driver_probe option looks good to me as the default.

I'm not sure how generic code can handle "multiple drivers".. Most devices will be able to work just fine with T=0 mode with bounce buffers so we should generally not encourage people to make completely different drivers for T=0/T=1 mode.

I think what is needed is some way for userspace to trigger the "locking configuration" you mentioned, that may need a special driver, but ONLY if the userspace is sequencing the device to T=1 mode. Not sure how to make that generic, but I think so long as userspace is explicitly controlling driver binding we can punt on that solution to the userspace project :)

Quoted 6 lines

The real nastyness is RAS - what do you do when the device falls out of RUN, the kernel driver should pretty much explode. But lots of people would like the kernel driver to stay alive and somehow we FLR, re-attest and "resume" the kernel driver without allowing any T=0 risks. For instance you can keep your netdev and just see a lot of lost packets while the driver thrashes.

Ideally the RUN->ERROR->UNLOCKED->LOCKED->RUN recovery can fit into the existing 'struct pci_error_handlers' regime in some farther out future.

It was a "fun" discovery to see that virtual AER injection does not exist in QEMU (at least last time I checked) and assigned devices that throw physical AER events just kill the VM.

Quoted 4 lines

But I think we can start with the idea that such RAS failures have to reload the driver too and work on improvements. Realistically few drivers have the sort of RAS features to consume this anyhow and maybe we introduce some "enhanced" driver mode to opt-into down the road.

Hmm, having trouble not reading that back supporting my argument above:

Realistically few devices support TDISP lets require enhanced drivers to opt-into TDISP for the time being.

#178 lore raw
Quoted 6 lines · On Fri, Aug 01, 2025 at 02:19:54PM -0700, [email protected] wrote:

On the host this establishes an SPDM session and sets up link encryption (IDE) with the physical device. Leave VMs out of the picture, this capability in isolation is a useful property. It addresses the similar threat model that Intel Total Memory Encryption (TME) or AMD Secure Memory Encryption (SME) go after, i.e. interposer on a physical link capturing data in flight.

Okay, maybe connect is not an intuitive name for opening IDE sessions..

Quoted 3 lines

I started this project with "all existing T=0 drivers 'just work'" as a goal and a virtue. I have been begrudgingly pulled away from it from the slow drip of complexity it appears to push into the PCI core.

Do you have some examples? I don't really see what complexity there is if the solution it simply not auto bind any drivers to TDISP capable devices and userspace is responsible to manually bind a driver once it has reached T=1.

This seems like the minimum possible simplicitly for the kernel as simply everything is managed by userspace, and there is really no special kernel behavior beyond switching the DMA API of an unbound driver on the T=0/1 change.

Quoted 2 lines

The concern is neither userspace nor the PCI core have everything it needs to get the device to T=1.

Disagree, I think userspace can have everything. It may need some per-device userspace support in difficult cases, but userspace can deal with it..

Quoted 2 lines

PCI core knows that the device is T=1 capable, but does not know how to preconfigure the device-specific lock state,

Userspace can do this. Can we define exactly what is needed to do this "pre-configure the device specific lock state"? At the very worst, for the most poorly designed device, userspace would have to bind a T=0 driver and then unbind it.

Again, I am trying to make something simple for the kernel that gets us to a working solution before we jump ahead to far more complex in the kernel models, like aware drivers that can toggle themselves between T=0/1.

Quoted 2 lines

Userspace might be able to bind a new driver that leaves the device in a lockable state on unbind, but that is not "just works" that is,

I wouldn't have the kernel leave the device in the locked state. That should always be userspace. The special driver may do whatever special setup is needed, then unbind and leave a normal unlocked device "prepped" for userspace locking without doing a FLR or something. Realistically I expect this to be a very rare requirement, I think this coming up just reflects the HW immaturity of some early TDISP devices.

Sensible mature devices should have no need of a pre-locking step. I think we should design toward that goal as the stable future and only try to enable a hacky work around for the problematic early devices. I certainly am not keen on seeing significant permanent kernel complexity to support this device design defect.

Quoted 5 lines

driver that expects the device arrives already running. Also, that main driver needs to be careful not to trigger typically benign actions like touch the command register to trip the device into ERROR state, or any device-specific actions that trip ERROR state but would otherwise be benign outside of TDISP."

As I said below, I disagree with this. You can't touch the *physical* command register but the cVM can certainly touch the *virtualized* command register. It up to the VMM To ensure this doesn't cause the device to fall out of RUN as part of virtualization.

I'd also say that the VMM should be responsible to set pBME=1 even if vBME=0? Shouldn't it? That simplifies even more things for the guest.

Quoted 7 lines
Quoted 4 lines

From that principal the kernel should NOT auto probe drivers to T=0 devices that can be made T=1. Userspace should handle attaching HW to such devices, and userspace can sequence whatever is required, including the attestation and verifying.

Agree, for PCI it would be simple to set a no-auto-probe policy for T=1 capable devices.

So then it is just a question of what does a userspace component need to do.

Quoted 5 lines

I do not want to burden the PCI core with TDISP compatibility hacks and workarounds if it turns out only a small handful of devices ever deploy a first generation TDISP Device Security Manager (DSM). L1 aiding L2, or TDISP simplicity improvements to allow the PCI core to handle this in a non-broken way, are what I expect if secure device assignment takes off.

Same feeling about pre-configuration :)

Quoted 5 lines
Quoted 3 lines

The starting point must have the core code do this sequence for every driver. Once that is working we can talk about if other flows are needed.

Do you agree that "device-specific-prep+lock" is the problem to solve?

Not "the" problem, but an design issue we need to accommodate but not endorse.

Quoted 9 lines
Quoted 4 lines

But I think we can start with the idea that such RAS failures have to reload the driver too and work on improvements. Realistically few drivers have the sort of RAS features to consume this anyhow and maybe we introduce some "enhanced" driver mode to opt-into down the road.

Hmm, having trouble not reading that back supporting my argument above:

Realistically few devices support TDISP lets require enhanced drivers to opt-into TDISP for the time being.

I would be comfortable if hitless RAS recovery for TDISP devices requires some kernel opt-in. But also I'm not sure how this should work from a security perspective. Should userspace also have to re-attest before allowing back to RUN? Clearly this is complicated.

Also, I would be comfortable to support this only for devices that do not require pre-configuration.

Jason

#179 lore raw
Quoted 11 lines · Jason Gunthorpe
Quoted 6 lines · On Fri, Aug 01, 2025 at 02:19:54PM -0700, [email protected] wrote:

On the host this establishes an SPDM session and sets up link encryption (IDE) with the physical device. Leave VMs out of the picture, this capability in isolation is a useful property. It addresses the similar threat model that Intel Total Memory Encryption (TME) or AMD Secure Memory Encryption (SME) go after, i.e. interposer on a physical link capturing data in flight.

Okay, maybe connect is not an intuitive name for opening IDE sessions..

Part of the rationale for a generic name is the TSM is free to assert that the link is secure without IDE. Think integrated devices where there is no expectation the link can be observed.

The host and guest side TSM operations are split into link/transport security and device/state security (private MMIO/DMA) concerns respectively. So maybe "secure_link" would be a better name for this host-side-only operation.

Quoted 8 lines
Quoted 3 lines

I started this project with "all existing T=0 drivers 'just work'" as a goal and a virtue. I have been begrudgingly pulled away from it from the slow drip of complexity it appears to push into the PCI core.

Do you have some examples? I don't really see what complexity there is if the solution it simply not auto bind any drivers to TDISP capable devices and userspace is responsible to manually bind a driver once it has reached T=1.

The example I have front of mind (confirmed by 2 vendors) is deferring the loading of guest-side device/state security capable firmware to the guest driver when the full device is assigned. In that scenario default device power-on firmware is capable of link/transport security, enough to get the device assigned. Guest needs to get the device/state security firmware loaded before TDISP state transitions are possible.

I do think RAS recovery needs it too, but like you say below that should come with conditions.

Quoted 11 lines

This seems like the minimum possible simplicitly for the kernel as simply everything is managed by userspace, and there is really no special kernel behavior beyond switching the DMA API of an unbound driver on the T=0/1 change.

Quoted 2 lines

The concern is neither userspace nor the PCI core have everything it needs to get the device to T=1.

Disagree, I think userspace can have everything. It may need some per-device userspace support in difficult cases, but userspace can deal with it..

I do think userspace can / must deal with it. Let me come back with actual patches and a sample test case. I see a potential path to support the above "prep" scenario without the mess of TDISP setup drivers, or the ugly complexity of driver toggles or a usermodehelper.

Quoted 8 lines
Quoted 2 lines

PCI core knows that the device is T=1 capable, but does not know how to preconfigure the device-specific lock state,

Userspace can do this. Can we define exactly what is needed to do this "pre-configure the device specific lock state"? At the very worst, for the most poorly designed device, userspace would have to bind a T=0 driver and then unbind it.

Quoted 4 lines

Again, I am trying to make something simple for the kernel that gets us to a working solution before we jump ahead to far more complex in the kernel models, like aware drivers that can toggle themselves between T=0/1.

Agree. When I talked about wishing for the simple TDISP case that is userspace can always "just lock" and "driver bind" without needing to worry about "prep", i.e any "prep" is always implied by "lock". That should be the baseline.

Quoted 11 lines
Quoted 2 lines

Userspace might be able to bind a new driver that leaves the device in a lockable state on unbind, but that is not "just works" that is,

I wouldn't have the kernel leave the device in the locked state. That should always be userspace. The special driver may do whatever special setup is needed, then unbind and leave a normal unlocked device "prepped" for userspace locking without doing a FLR or something. Realistically I expect this to be a very rare requirement, I think this coming up just reflects the HW immaturity of some early TDISP devices.

Quoted 5 lines

Sensible mature devices should have no need of a pre-locking step. I think we should design toward that goal as the stable future and only try to enable a hacky work around for the problematic early devices. I certainly am not keen on seeing significant permanent kernel complexity to support this device design defect.

Yeah, that is the nightmare I had last night. I completed the thought exercise about driver toggle and said, "whoops, nope, Jason is right, we can't design for that without leaving a permanent mess to cleanup". The end goal needs to look like straight line typical driver probe path for TDISP capable devices.

Quoted 6 lines
Quoted 5 lines

driver that expects the device arrives already running. Also, that main driver needs to be careful not to trigger typically benign actions like touch the command register to trip the device into ERROR state, or any device-specific actions that trip ERROR state but would otherwise be benign outside of TDISP."

Quoted 7 lines

As I said below, I disagree with this. You can't touch the *physical* command register but the cVM can certainly touch the *virtualized* command register. It up to the VMM To ensure this doesn't cause the device to fall out of RUN as part of virtualization.

I'd also say that the VMM should be responsible to set pBME=1 even if vBME=0? Shouldn't it? That simplifies even more things for the guest.

True. Although, now I am going back on my PCI core burden concern to wonder if *it* should handle a vBME on behalf of the driver if only because it may want to force the device out of the RUN state on driver unbind to meet typical pci_disable_device() expectations.

Alexey had this, I thought it was burdensome, now coming around.

Quoted 19 lines
Quoted 7 lines
Quoted 4 lines

From that principal the kernel should NOT auto probe drivers to T=0 devices that can be made T=1. Userspace should handle attaching HW to such devices, and userspace can sequence whatever is required, including the attestation and verifying.

Agree, for PCI it would be simple to set a no-auto-probe policy for T=1 capable devices.

So then it is just a question of what does a userspace component need to do.

Quoted 5 lines

I do not want to burden the PCI core with TDISP compatibility hacks and workarounds if it turns out only a small handful of devices ever deploy a first generation TDISP Device Security Manager (DSM). L1 aiding L2, or TDISP simplicity improvements to allow the PCI core to handle this in a non-broken way, are what I expect if secure device assignment takes off.

Same feeling about pre-configuration :)

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

The starting point must have the core code do this sequence for every driver. Once that is working we can talk about if other flows are needed.

Do you agree that "device-specific-prep+lock" is the problem to solve?

Not "the" problem, but an design issue we need to accommodate but not endorse.

I hear you, let me walk back from the cliff with patches.

Quoted 11 lines
Quoted 10 lines
Quoted 5 lines

But I think we can start with the idea that such RAS failures have to reload the driver too and work on improvements. Realistically few drivers have the sort of RAS features to consume this anyhow and maybe we introduce some "enhanced" driver mode to opt-into down the road.

Hmm, having trouble not reading that back supporting my argument above:

Realistically few devices support TDISP lets require enhanced drivers to opt-into TDISP for the time being.

Quoted 7 lines

I would be comfortable if hitless RAS recovery for TDISP devices requires some kernel opt-in. But also I'm not sure how this should work from a security perspective. Should userspace also have to re-attest before allowing back to RUN? Clearly this is complicated.

Also, I would be comfortable to support this only for devices that do not require pre-configuration.

That seems reasonable. You want hitless RAS? Give us hitless init.

#180 lore raw
Quoted 11 lines · On Sat, Aug 02, 2025 at 04:50:50PM -0700, [email protected] wrote:
Quoted 4 lines

Do you have some examples? I don't really see what complexity there is if the solution it simply not auto bind any drivers to TDISP capable devices and userspace is responsible to manually bind a driver once it has reached T=1.

The example I have front of mind (confirmed by 2 vendors) is deferring the loading of guest-side device/state security capable firmware to the guest driver when the full device is assigned. In that scenario default device power-on firmware is capable of link/transport security, enough to get the device assigned. Guest needs to get the device/state security firmware loaded before TDISP state transitions are possible.

Yeah, those are the only cases I know of too, and IMHO, they are just early devices. Clearly the clean answer is to put enough boot FW on the device's flash to get to T=1 mode, then have the trusted OS driver load the operating firmware from the trusted OS filesystem though the trusted bootloader T=1 device.

You effectively attest the bootloader, and then if you trust the bootloader you know that when the device gets to T=1 it can be trusted to properly run the FW the trusted driver provides.

Think about this more broadly, does the prep FW load idea make sense for something like SRIOV? No, it really doesn't. The hypervisor loaded FW that is running the PF should definately be strong enough to get to T=1 on the VM/VF side as well.

The non-SRIOV cases are quite often whole machine assignment scenarios. But I'm sensing alot of that space is moving toward bare metal machines instead of VMs.

I wonder if you can use all the CC machinery to attest and secure a bare metal host?

Quoted 2 lines

I do think RAS recovery needs it too, but like you say below that should come with conditions.

Especially RAS becomes simple because it basically follows the normal flows that existed prior to TDISP, with the exception of needing some attestation step.

I don't know alot about CC attestation, but maybe we can have userspace provide the kernel with the accepted measurement and then for RAS the kernel can FLR, remeasure and if the measurement is exactly the same go back into T=1 automatically as part of the PCI core FLR logic.

Quoted 4 lines

I do think userspace can / must deal with it. Let me come back with actual patches and a sample test case. I see a potential path to support the above "prep" scenario without the mess of TDISP setup drivers, or the ugly complexity of driver toggles or a usermodehelper.

I don't see how, something nasty has to be done in the kernel to allow an attached driver to switch between T=1 and T=0 "views" of the device and lockstep those changes with userspace. This is not so simple and it really basically exactly the same as driver binding.

I don't think we should be afraid of T=0 prep drivers in these early days.

Something more complex could come later if it is really warranted and people really insist on continuing this unclean device design strategy.

Quoted 5 lines

Yeah, that is the nightmare I had last night. I completed the thought exercise about driver toggle and said, "whoops, nope, Jason is right, we can't design for that without leaving a permanent mess to cleanup". The end goal needs to look like straight line typical driver probe path for TDISP capable devices.

Yeah, maybe it is worthwhile to someday try to figure out an alternative - keep in mind that critically this requires someone to also come with an intree driver that will use all these new APIs and capabilities!!!

So lets get walking first and then someone can come with some proposal, complete with a driver implementing it, and it can be judged. This project is already so big, and I'm pretty sure if you start to also need entirely new operating modes for drivers the basics will just get bogged down in that discussion, and very likely killed anyhow due to a lack of user.

Even if we decide that is prefered it is better to separate it and discuss it after the basics are merged. At least where I sit getting basic guest support is a big priority so I strongly want to strip it down to minimal as possible to make consistent progress steps.

Quoted 4 lines

True. Although, now I am going back on my PCI core burden concern to wonder if *it* should handle a vBME on behalf of the driver if only because it may want to force the device out of the RUN state on driver unbind to meet typical pci_disable_device() expectations.

Hiding some vBME in the PCI core might make sense if we can't get the VMM owners to agree to do it on the hypervisor side. It works better on the VMM side because there is always an IOMMU and the VMM can emulate BME by blocking DMA with the IOMMU.

But I would not allow/expect kernel device drivers to have anything to do with the TDISP states. Getting into RUN is fully sequenced by userspace, getting out of run should also be sequenced only by userspace.

Removing a driver does not change the trust state of the PCI device, so it shouldn't drop out of RUN. If userspace wishes to FLR the device after userspace asked to unbind it can, there are already sysfs controls for this IIRC.

Basically, all this says that Linux drivers that want to be used with T=1 should be well behaved, fully quite all their DMA on remove, and have no *functional* need for BME to do anyhting. We pretty much already expect this of drivers today, so I don't see an issue with strongly requiring it for T=1.

Keep in mind the flip side, almost no drivers are structured properly to forcibly quiet any DMA before pci_enable_device(). Some HW, like mlx5, can't do this at all without either using DMA to send a reset command or through FLR.

Quoted 9 lines
Quoted 7 lines

I would be comfortable if hitless RAS recovery for TDISP devices requires some kernel opt-in. But also I'm not sure how this should work from a security perspective. Should userspace also have to re-attest before allowing back to RUN? Clearly this is complicated.

Also, I would be comfortable to support this only for devices that do not require pre-configuration.

That seems reasonable. You want hitless RAS? Give us hitless init.

Yeah.. Realistically there are few drivers that can even do this today, mlx5 for example has such code (and it is hard!).

There is alot of investment required in the driver's core subsystem to make this work. netdev and RDMA can support a 'rebirth' sort of flow where the driver can disconnect the SW APIs, FLR the device, then reconnect in some way. However, for example, I recently had a discussion with DRM guys about RAS and they are not even doing the basic locking/etc to be able to do this. :\

Jason

author maintainer
#181 lore raw
In reply to [email protected]
Quoted 75 lines · <[email protected]> writes:
Quoted 15 lines · Jason Gunthorpe
Quoted 14 lines · On Thu, Jul 31, 2025 at 07:07:17PM -0700, [email protected] wrote:
Quoted 8 lines · Aneesh Kumar K.V (Arm)

Host:
step 1.
echo ${DEVICE} > /sys/bus/pci/devices/${DEVICE}/driver/unbind echo vfio-pci > /sys/bus/pci/devices/${DEVICE}/driver_override echo ${DEVICE} > /sys/bus/pci/drivers_probe

step 2.
echo 1 > /sys/bus/pci/devices/$DEVICE/tsm/connect

Just for my own understanding... presumably there is no ordering constraint for ARM CCA between step1 and step2, right? I.e. The connect state is independent of the bind state.

Quoted 7 lines
Quoted 3 lines

In the v4 PCI/TSM scheme the connect command is now:

echo $tsm_dev > /sys/bus/pci/devices/$DEVICE/tsm/connect

What does this do on the host? It seems to somehow prep it for VM assignment? Seems pretty strange this is here in sysfs and not part of creating the vPCI function in the VM through VFIO and iommufd?

vPCI is out of the picture at this phase.

On the host this establishes an SPDM session and sets up link encryption (IDE) with the physical device. Leave VMs out of the picture, this capability in isolation is a useful property. It addresses the similar threat model that Intel Total Memory Encryption (TME) or AMD Secure Memory Encryption (SME) go after, i.e. interposer on a physical link capturing data in flight.

With that established then one can go futher to do the full TDISP dance.

Quoted 5 lines

Frankly, I'm nervous about making any uAPI whatsoever for the hypervisor side at this point. I don't think we have enough of the solution even in draft format. I'd really like your first merged TSM series to only have uAPI for the guest side where things are hopefully closer to complete.

Aligned. I am not comfortable merging any of this until we have that end to end reliably stable for a kernel cycle or 2. The proposal is soak all the vendor solutions together in tsm.git#staging.

Now, if the guest side graduates out of that staging before the host side, I am ok with that.

Quoted 17 lines
Quoted 9 lines
Quoted 5 lines

step 1:
echo ${DEVICE} > /sys/bus/pci/devices/${DEVICE}/driver/unbind

step 2: Move the device to TDISP LOCK state
echo 1 > /sys/bus/pci/devices/${DEVICE}/tsm/lock

Ok, so my stance has recently picked up some nuance here. As Jason mentions here:

Quoted 7 lines

http://lore.kernel.org/[email protected]

"However it works, it should be done before the driver is probed and remain stable for the duration of the driver attachment. From the iommu side the correct iommu domain, on the correct IOMMU instance to handle the expected traffic should be setup as the DMA API's iommu domain."

Quoted 3 lines

I think it is not just the dma api, but also the MMIO registers may move location (form shared to protected IPA space for example). Meaning any attached driver is completely wrecked.

True.

Quoted 4 lines
Quoted 3 lines

I agree with that up until the point where the implication is userspace control of the UNLOCKED->LOCKED transition. That transition requires enabling bus-mastering (BME),

Quoted 8 lines
Quoted 3 lines

Why? That's sad. BME should be controlled by the VM driver not the TSM, and it should be set only when a VM driver is probed to the RUN state device?

To me it is an unfortunate PCI specification wrinkle that writing to the command register drops the device from RUN to ERROR. So you can LOCK without setting BME, but then no DMA.

This is only w.r.t clearing BME isn't ?

According to section 11.2.6 DSM Tracking and Handling of Locked TDI Configurations

Clearing any of the following bits causes the TDI hosted by the Function to transition to ERROR:

• Memory Space Enable
• Bus Master Enable

Which implies the flow described in the cover-letter where driver enable the BME works? However clearing BME may be problematic? I did have a FIXME!!/comment in [1]

vfio_pci_core_close_device():

#if 0
	/*
	 * destroy vdevice which involves tsm unbind before we disable pci disable
	 * A MSE/BME clear will transition the device to error state.
	 */
	if (core_vdev->iommufd_device)
		iommufd_device_tombstone_vdevice(core_vdev->iommufd_device);

#endif

	vfio_pci_core_disable(vdev);

Currently, we destroy (TSM unbind) the vdevice after calling vfio_pci_core_disable(), which means BME is cleared before unbinding, and the TDI transitions to the ERROR state.

[1] https://lore.kernel.org/all/[email protected]/

-aneesh

#182 lore raw
Quoted 6 lines · Aneesh Kumar K.V
Quoted 3 lines

To me it is an unfortunate PCI specification wrinkle that writing to the command register drops the device from RUN to ERROR. So you can LOCK without setting BME, but then no DMA.

This is only w.r.t clearing BME isn't ?

Quoted 7 lines

According to section 11.2.6 DSM Tracking and Handling of Locked TDI Configurations

Clearing any of the following bits causes the TDI hosted by the Function to transition to ERROR:

• Memory Space Enable
• Bus Master Enable

Oh that's nice, yeah!

Quoted 12 lines

Which implies the flow described in the cover-letter where driver enable the BME works? However clearing BME may be problematic? I did have a FIXME!!/comment in [1]

vfio_pci_core_close_device():

#if 0
	/*
	 * destroy vdevice which involves tsm unbind before we disable pci disable
	 * A MSE/BME clear will transition the device to error state.
	 */
	if (core_vdev->iommufd_device)
		iommufd_device_tombstone_vdevice(core_vdev->iommufd_device);
Quoted 3 lines

#endif

	vfio_pci_core_disable(vdev);

Here is where I feel the VMM should be trapping this and NOPing it, or failing that the guest PCI Core should NOP it.

With the ideal version being the TSM and VMM would be able to block the iommu as a functional stand in for BME.

Quoted 3 lines

Currently, we destroy (TSM unbind) the vdevice after calling vfio_pci_core_disable(), which means BME is cleared before unbinding, and the TDI transitions to the ERROR state.

I don't think this ordering is deliberate, we can destroy the vdevice much earlier??

Jason

#183 lore raw
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · Aneesh Kumar K.V
Quoted 3 lines

To me it is an unfortunate PCI specification wrinkle that writing to the command register drops the device from RUN to ERROR. So you can LOCK without setting BME, but then no DMA.

This is only w.r.t clearing BME isn't ?

According to section 11.2.6 DSM Tracking and Handling of Locked TDI Configurations

Quoted 7 lines
Quoted 5 lines

Clearing any of the following bits causes the TDI hosted by the Function to transition to ERROR:

• Memory Space Enable
• Bus Master Enable

Oh that's nice, yeah!

That is useful, but an unmodified PCI driver is going to make separate calls to pci_set_master() and pci_enable_device() so it should still be the case that those need to be trapped out of the concern that writing back zero for a read-modify-write also trips the error state on some device that fails the Robustness Principle.

I guess we could wait to solve that problem until the encountering the first device that trips ERROR when writing zero to an already zeroed bit.

Quoted 14 lines
Quoted 14 lines

Which implies the flow described in the cover-letter where driver enable the BME works? However clearing BME may be problematic? I did have a FIXME!!/comment in [1]

vfio_pci_core_close_device():

#if 0
	/*
	 * destroy vdevice which involves tsm unbind before we disable pci disable
	 * A MSE/BME clear will transition the device to error state.
	 */
	if (core_vdev->iommufd_device)
		iommufd_device_tombstone_vdevice(core_vdev->iommufd_device);

#endif

Quoted 4 lines
Quoted 1 line
	vfio_pci_core_disable(vdev);

Here is where I feel the VMM should be trapping this and NOPing it, or failing that the guest PCI Core should NOP it.

At this point (vfio shutdown path) the VMM is committed stopping guest operations with the device. So ok not to not NOP in this specific path, right?

Quoted 2 lines

With the ideal version being the TSM and VMM would be able to block the iommu as a functional stand in for BME.

The TSM block for BME is the LOCKED or ERROR state. That would be in conflict with the proposal that the device stays in the RUN state on guest driver unbind.

I feel like either the device stays in RUN state and BME leaks, or the device is returned to LOCKED on driver unbind. Otherwise a functional stand-in for BME that also keeps the device in RUN state feels like a TSM feature request for a "RUN but BLOCKED" state.

#184 lore raw
Quoted 6 lines · On Tue, Aug 05, 2025 at 11:27:36AM -0700, [email protected] wrote:
Quoted 6 lines
Quoted 5 lines

Clearing any of the following bits causes the TDI hosted by the Function to transition to ERROR:

• Memory Space Enable
• Bus Master Enable

Quoted 7 lines
Quoted 1 line

Oh that's nice, yeah!

That is useful, but an unmodified PCI driver is going to make separate calls to pci_set_master() and pci_enable_device() so it should still be the case that those need to be trapped out of the concern that writing back zero for a read-modify-write also trips the error state on some device that fails the Robustness Principle.

I hope we don't RMW BME and MSE in some weird way like that :(

Quoted 6 lines
Quoted 2 lines

Here is where I feel the VMM should be trapping this and NOPing it, or failing that the guest PCI Core should NOP it.

At this point (vfio shutdown path) the VMM is committed stopping guest operations with the device. So ok not to not NOP in this specific path, right?

What I said in my other mail was the the T=1 state should have nothing to do with driver binding. So unbinding vfio should leave the device in the RUN state just fine.

Quoted 6 lines
Quoted 2 lines

With the ideal version being the TSM and VMM would be able to block the iommu as a functional stand in for BME.

The TSM block for BME is the LOCKED or ERROR state. That would be in conflict with the proposal that the device stays in the RUN state on guest driver unbind.

This is a different thing. Leaving RUN says the OS (especially userspace) does not trust the device.

Disabling DMA, on explict trusted request from the cVM, is entirely fine to do inside the T=1 state. PCI made it so the only way to do this is with the IOMMU, oh well, so be it.

Quoted 2 lines

I feel like either the device stays in RUN state and BME leaks, or the device is returned to LOCKED on driver unbind.

Stay in RUN is my vote. I can't really defend the other choice from a linux driver model perspective.

Quoted 3 lines

Otherwise a functional stand-in for BME that also keeps the device in RUN state feels like a TSM feature request for a "RUN but BLOCKED" state.

Yes, and probably not necessary, more of a defence against bugs in depth kind of request. For Linux we would like it if the device can be in RUN and have DMA blocked off during all times when no driver is attached.

Jason

#185 lore raw
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · On Tue, Aug 05, 2025 at 11:27:36AM -0700, [email protected] wrote:
Quoted 7 lines
Quoted 5 lines

Clearing any of the following bits causes the TDI hosted by the Function to transition to ERROR:

• Memory Space Enable
• Bus Master Enable

Oh that's nice, yeah!

Quoted 7 lines
Quoted 5 lines

That is useful, but an unmodified PCI driver is going to make separate calls to pci_set_master() and pci_enable_device() so it should still be the case that those need to be trapped out of the concern that writing back zero for a read-modify-write also trips the error state on some device that fails the Robustness Principle.

I hope we don't RMW BME and MSE in some weird way like that :(

Yeah, I would like to say, "device, you get to keep the pieces if you transition to ERROR state on re-writing on already zeroed-bit."

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

Here is where I feel the VMM should be trapping this and NOPing it, or failing that the guest PCI Core should NOP it.

At this point (vfio shutdown path) the VMM is committed stopping guest operations with the device. So ok not to not NOP in this specific path, right?

What I said in my other mail was the the T=1 state should have nothing to do with driver binding.

Guest driver unbind, agree.

Quoted 1 line

So unbinding vfio should leave the device in the RUN state just fine.

Perhaps my vfio inexperience is showing, but at the point where the VMM is unbinding vfio it is committed to destroying the guest's assigned device context, no? So should that not be the point where continuing to maintain the RUN state ends?

Quoted 20 lines
Quoted 6 lines
Quoted 2 lines

With the ideal version being the TSM and VMM would be able to block the iommu as a functional stand in for BME.

The TSM block for BME is the LOCKED or ERROR state. That would be in conflict with the proposal that the device stays in the RUN state on guest driver unbind.

This is a different thing. Leaving RUN says the OS (especially userspace) does not trust the device.

Disabling DMA, on explict trusted request from the cVM, is entirely fine to do inside the T=1 state. PCI made it so the only way to do this is with the IOMMU, oh well, so be it.

Quoted 2 lines

I feel like either the device stays in RUN state and BME leaks, or the device is returned to LOCKED on driver unbind.

Stay in RUN is my vote. I can't really defend the other choice from a linux driver model perspective.

Quoted 8 lines
Quoted 3 lines

Otherwise a functional stand-in for BME that also keeps the device in RUN state feels like a TSM feature request for a "RUN but BLOCKED" state.

Yes, and probably not necessary, more of a defence against bugs in depth kind of request. For Linux we would like it if the device can be in RUN and have DMA blocked off during all times when no driver is attached.

Ok, defense in depth, but in the meantime rely on unbound driver == DMA unmapped and device should be quiescent. Combine that with the fact that userspace PCI drivers should be disabled in cVMs should mean that guest can expect that an unbound TDI in the RUN state will remain quiet.

#186 lore raw
Quoted 6 lines · On Tue, Aug 05, 2025 at 12:06:11PM -0700, [email protected] wrote:
Quoted 1 line

So unbinding vfio should leave the device in the RUN state just fine.

Perhaps my vfio inexperience is showing, but at the point where the VMM is unbinding vfio it is committed to destroying the guest's assigned device context, no? So should that not be the point where continuing to maintain the RUN state ends?

Oh, sorry it gets so confusing..

VFIO *in the guest* should behave as above, like any other driver unbind leaves it in RUN.

VFIO *in the host* should leave the RUN state at the soonest of:

- cVM's KVM is destroyed
- iommufd vdevice is destroyed
- vfio device is closed

And maybe more cases I didn't think of.. BME should happen strictly after all of the above and should not be the trigger that drops it out of RUN.

Quoted 9 lines
Quoted 4 lines

Yes, and probably not necessary, more of a defence against bugs in depth kind of request. For Linux we would like it if the device can be in RUN and have DMA blocked off during all times when no driver is attached.

Ok, defense in depth, but in the meantime rely on unbound driver == DMA unmapped and device should be quiescent. Combine that with the fact that userspace PCI drivers should be disabled in cVMs should mean that guest can expect that an unbound TDI in the RUN state will remain quiet.

"userspace PCI drivers" is VFIO in the guest which means you get FLRs to fence the DMA.

If we end up where I suggested earlier for RAS that a FLR can check the attestation and if exactly matching reaccept it automatically then it would maintain the 'once accepted we stay in T=1 RUN state' idea.

Jason

author maintainer
#187 lore raw
In reply to [email protected]
Quoted 23 lines · <[email protected]> writes:
Quoted 9 lines · Aneesh Kumar K.V (Arm)

This patch series implements support for Device Assignment in the ARM CCA architecture. The code changes are based on Alp12 specification published here [1].

The code builds on the TSM framework patches posted at [2]. We add extension to that framework so that TSM is now used in both the host and the guest.

A DA workflow can be summarized as below:

Quoted 8 lines

Host:
step 1.
echo ${DEVICE} > /sys/bus/pci/devices/${DEVICE}/driver/unbind echo vfio-pci > /sys/bus/pci/devices/${DEVICE}/driver_override echo ${DEVICE} > /sys/bus/pci/drivers_probe

step 2.
echo 1 > /sys/bus/pci/devices/$DEVICE/tsm/connect

Just for my own understanding... presumably there is no ordering constraint for ARM CCA between step1 and step2, right? I.e. The connect state is independent of the bind state.

Quoted 9 lines

In the v4 PCI/TSM scheme the connect command is now:

echo $tsm_dev > /sys/bus/pci/devices/$DEVICE/tsm/connect

Quoted 1 line

Now in the guest we follow the below steps

I assume a signifcant amount of kvmtool magic happens here to get the TDI into a "bind capable" state, can you share that command?

lkvm run --realm -c 2 -m 256 -k /kselftest/Image -p "$KERNEL_PARAMS" -d ./rootfs-guest.ext2 --iommufd-vdevice --vfio-pci $DEVICE1 --vfio-pci $DEVICE2

Quoted 8 lines

I had been assuming that everyone was prototyping with QEMU. Not a problem per se, but the memory management for shared device assignment / bounce buffering has had a quite of bit of work on the QEMU side, so just curious about the difference in approach here. Like, does kvmtool support operating the device in shared mode with bounce buffering and page conversion (shared <=> private) support? In any event, happy to see mutiple simultaneous consumers of this new kernel infrastructure.

-aneesh