ARM CCA Device Assignment support
Subsystems (13)
Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report
In reply to Arto MerilainenQuoted 15 lines · Arto Merilainen
Quoted 13 lines · Aneesh Kumar K.V (Arm)
drivers/virt/coco/arm-cca-guest/rsi-da.c +13
-0+for (int i =0 ; i < interface_report->mmio_range_count; i++, mmio_range++) { + +/*FIXME!! units in 4K size*/ + range_id =FIELD_GET (TSM_INTF_REPORT_MMIO_RANGE_ID , mmio_range->range_attributes); + +/* no secure interrupts */ +if (msix_tbl_bar !=-1 && range_id == msix_tbl_bar) { +pr_info ("Skipping misx table\n" ); +continue ; + } + +if (msix_pba_bar !=-1 && range_id == msix_pba_bar) { +pr_info ("Skipping misx pba\n" );
Quoted 8 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 292
Quoted 3 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +3
-0+continue ; + } +MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated.
Right, there are not enough BARs in most devices to give MSI its own BAR.
Quoted 4 lines
It is enough to have sufficient isolation within the
BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.
Right, this sounds bad.
Quoted 3 lines
Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?
IMHO this is a mess. The virtualization must end up putting a shared page(s) covering the MSI space in the middle of the MMIO region.
I think this should be done by fragmenting the layout in the IPA where the private MMIO is within the protected IPA space with an unmapped hole covering the MSIX registers. The acceptance process should validate this.
The MSIX registers would then be located in the shared IPA space.
A normal driver mmaping it's BAR will then crash if it tries to access the MSIX registers. This is good, we want to catch these non-secure configurations and block them.
The MSI code will have to know to compute the shared IPA alias and use that.
Jason
Aneesh Kumar K.V
maintainerRe: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report
In reply to Arto MerilainenQuoted 27 lines · Arto Merilainen
Quoted 6 lines · Aneesh Kumar K.V (Arm)
drivers/virt/coco/arm-cca-guest/rsi-da.c +6
-0+for (int i =0 ; i < interface_report->mmio_range_count; i++, mmio_range++) { + +/*FIXME!! units in 4K size*/ + range_id =FIELD_GET (TSM_INTF_REPORT_MMIO_RANGE_ID , mmio_range->range_attributes); + +/* no secure interrupts */ drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 297
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+if (msix_tbl_bar !=-1 && range_id == msix_tbl_bar) { +pr_info ("Skipping misx table\n" ); +continue ; + } + +if (msix_pba_bar !=-1 && range_id == msix_pba_bar) { +pr_info ("Skipping misx pba\n" ); +continue ; + } +MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated. It is enough to have sufficient isolation within the BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.
Quoted 4 lines
Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?
Yes, that was added because at one point the FVP model was including the MSI-X table and PBA regions in the interface report.
If I understand correctly, we shouldn't expect to see those regions in the report unless secure interrupts are supported. The BAR-based skipping was added as a workaround to handle the FVP issue.
I believe we can drop that workaround now—if those regions are incorrectly present, the below validation logic should catch and reject them appropriately. Does that sound reasonable?
/* No secure interrupts, we should not find this set, ignore for now. */
if (FIELD_GET(TSM_INTF_REPORT_MMIO_MSIX_TABLE, mmio_range->range_attributes) ||
FIELD_GET(TSM_INTF_REPORT_MMIO_PBA, mmio_range->range_attributes)) {
pci_info(pdev, "Skipping MSIX (%ld/%ld) range [%d] #%d %d pages, %llx..%llx\n",
FIELD_GET(TSM_INTF_REPORT_MMIO_MSIX_TABLE, mmio_range->range_attributes),
FIELD_GET(TSM_INTF_REPORT_MMIO_PBA, mmio_range->range_attributes),
i, range_id, mmio_range->num_pages, r->start, r->end);
continue;
}-aneesh
Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report
Quoted 29 lines · Aneesh Kumar K.V
Quoted 27 lines · Arto Merilainen
Quoted 6 lines · Aneesh Kumar K.V (Arm)
drivers/virt/coco/arm-cca-guest/rsi-da.c +6
-0+for (int i =0 ; i < interface_report->mmio_range_count; i++, mmio_range++) { + +/*FIXME!! units in 4K size*/ + range_id =FIELD_GET (TSM_INTF_REPORT_MMIO_RANGE_ID , mmio_range->range_attributes); + +/* no secure interrupts */ drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 297
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+if (msix_tbl_bar !=-1 && range_id == msix_tbl_bar) { +pr_info ("Skipping misx table\n" ); +continue ; + } + +if (msix_pba_bar !=-1 && range_id == msix_pba_bar) { +pr_info ("Skipping misx pba\n" ); +continue ; + } +MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated. It is enough to have sufficient isolation within the BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.
Quoted 7 lines
Quoted 4 lines
Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?
Yes, that was added because at one point the FVP model was including the MSI-X table and PBA regions in the interface report.
The issue I am raising is a different one. The MSI-X table and PBA may reside in the middle of a BAR range, and the BAR range may contain registers which should be accessible only via protected IPA. In this case I would expect the pages before and after the MSI-X table (and PBA) to be validated while the pages related to MSI-X structures would be left unprotected.
Given that the MSI-X table or PBA regions shouldn't be present in the interface report, the code needs to first find out where the MSI-X structures reside, and use this information to determine which "sub-ranges" should be skipped over during validation.
Quoted 4 lines
If I understand correctly, we shouldn't expect to see those regions in the report unless secure interrupts are supported. The BAR-based skipping was added as a workaround to handle the FVP issue.
Correct. Assuming that RMM doesn't lock the MSI-X table, I'd expect these regions to be omitted in the interface report.
Quoted 4 lines
I believe we can drop that workaround now—if those regions are incorrectly present, the below validation logic should catch and reject them appropriately. Does that sound reasonable?
Quoted 11 lines
/* No secure interrupts, we should not find this set, ignore for now. */
if (FIELD_GET(TSM_INTF_REPORT_MMIO_MSIX_TABLE, mmio_range->range_attributes) ||
FIELD_GET(TSM_INTF_REPORT_MMIO_PBA, mmio_range->range_attributes)) {
pci_info(pdev, "Skipping MSIX (%ld/%ld) range [%d] #%d %d pages, %llx..%llx\n",
FIELD_GET(TSM_INTF_REPORT_MMIO_MSIX_TABLE, mmio_range->range_attributes),
FIELD_GET(TSM_INTF_REPORT_MMIO_PBA, mmio_range->range_attributes),
i, range_id, mmio_range->num_pages, r->start, r->end);
continue;
}First, I think this is skipping over the whole range => if there are registers outside of the MSI-X structures (but within the same BAR), they won't be validated.
Second, if the MSI-X regions are present in the interface report, wouldn't it - in the common case - mean that the device expects the ranges to be accessed with T=1? If this happens unexpectedly, it sounds that MSI-X wouldn't be usable. I wonder if the code should simply return an error instead if informing the user via pci_info()...
- R2
Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report
In reply to Arto MerilainenQuoted 24 lines · Arto Merilainen
Quoted 8 lines · Aneesh Kumar K.V (Arm)
+ for (int i = 0; i <
interface_report->mmio_range_count; i++, mmio_range++) {
+
+ /*FIXME!! units in 4K size*/
+range_id=FIELD_GET(TSM_INTF_REPORT_MMIO_RANGE_ID, mmio_range->range_attributes);drivers/virt/coco/arm-cca-guest/rsi-da.c +2
-0+ + /* no secure interrupts */drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 297
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+ if (msix_tbl_bar != -1 && range_id == msix_tbl_bar) { + pr_info("Skipping misx table\n"); + continue; + } + + if (msix_pba_bar != -1 && range_id == msix_pba_bar) { + pr_info("Skipping misx pba\n"); + continue; + } +MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated. It is enough to have sufficient isolation within the BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.
Quoted 2 lines
Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?
I re-reviewed my suggestion, and what I proposed here seems wrong. However, I think there is a different more generic problem related to the MSI-X table, PBA and non-TEE ranges.
If a BAR is sparse (e.g., it has TEE pages and the MSI-X table, PBA or non-TEE areas), the TDISP interface report may contain multiple ranges with the same range_id (/BAR id). In case a BAR contains some registers in low addresses, the MSI-X table and other registers after the MSI-X table, the interface report is expected to have two ranges for the same BAR with different "first 4k page" and "size" fields.
This creates a tricky problem given that RSI_VDEV_VALIDATE_MAPPING requires both the ipa_base and pa_base which should correspond to the same location. In above scenario, the PA of the first range would correspond to the BAR base whereas the second range would correspond to a location residing after the MSI-X table.
Assuming that the report contains obfuscated (but linear) physical addresses, it would be possible to create heuristics for this case. However, the fundamental problem is that none of the "first 4k page" fields in the ranges is guaranteed to correspond to the base of any BAR: Consider a case where the MSI-X table is in the beginning of a BAR and it is followed by a single TEE range. If the MSI-X is not locked, the "first 4k page" field will not correspond to the beginning of the BAR. If the realm naiviely reads the ipa_base using pci_resouce_n() and corresponding pa_base from the interface report, the addresses won't match and the validation will fail.
It seems that interpreting the interface report cannot be done without knowledge of the device's register layout. Therefore, I don't think the ranges can be validated/remapped automatically without involving the device driver, but there should be APIs for reading the interface report, and for requesting making specific ranges protected.
- R2
Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report
Quoted 5 lines · Arto Merilainen
This creates a tricky problem given that
RSI_VDEV_VALIDATE_MAPPINGrequires both theipa_baseandpa_basewhich should correspond to the same location. In above scenario, the PA of the first range would correspond to the BAR base whereas the second range would correspond to a location residing after the MSI-X table.
This seems like a defect in the RSI_VDEV_VALIDATE_MAPPING - it should be able to consume the same format of data that the tdisp report emits to validate it.
From a kernel side we also should be careful that the driver isn't tricked into mapping MMIO that is not secure when it should be. Presumably all the default io access functions should demand secure memory in T=1 mode, and special ones like the MSI-X code would have some special version to accept either?
Jason
Aneesh Kumar K.V
maintainerRe: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report
In reply to Arto MerilainenQuoted 62 lines · Arto Merilainen
Quoted 29 lines · Arto Merilainen
Quoted 8 lines · Aneesh Kumar K.V (Arm)
+ for (int i = 0; i <
interface_report->mmio_range_count; i++, mmio_range++) {
+
+ /*FIXME!! units in 4K size*/
+range_id=FIELD_GET(TSM_INTF_REPORT_MMIO_RANGE_ID, mmio_range->range_attributes);drivers/virt/coco/arm-cca-guest/rsi-da.c +2
-0+ + /* no secure interrupts */drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 297
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+ if (msix_tbl_bar != -1 && range_id == msix_tbl_bar) { + pr_info("Skipping misx table\n"); + continue; + } + + if (msix_pba_bar != -1 && range_id == msix_pba_bar) { + pr_info("Skipping misx pba\n"); + continue; + } +MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated. It is enough to have sufficient isolation within the BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.
Quoted 3 lines
Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?
I re-reviewed my suggestion, and what I proposed here seems wrong. However, I think there is a different more generic problem related to the MSI-X table, PBA and non-TEE ranges.
If a BAR is sparse (e.g., it has TEE pages and the MSI-X table, PBA or non-TEE areas), the TDISP interface report may contain multiple ranges with the same
range_id(/BAR id). In case a BAR contains some registers in low addresses, the MSI-X table and other registers after the MSI-X table, the interface report is expected to have two ranges for the same BAR with different "first 4k page" and "size" fields.This creates a tricky problem given that
RSI_VDEV_VALIDATE_MAPPINGrequires both theipa_baseandpa_basewhich should correspond to the same location. In above scenario, the PA of the first range would correspond to the BAR base whereas the second range would correspond to a location residing after the MSI-X table.Assuming that the report contains obfuscated (but linear) physical addresses, it would be possible to create heuristics for this case. However, the fundamental problem is that none of the "first 4k page" fields in the ranges is guaranteed to correspond to the base of any BAR:
Consider a case where the MSI-X table is in the beginning of a BAR and it is followed by a single TEE range. If the MSI-X is not locked, the "first 4k page" field will not correspond to the beginning of the BAR. If the realm naiviely reads theipa_baseusingpci_resouce_n()and correspondingpa_basefrom the interface report, the addresses won't match and the validation will fail.
Quoted 6 lines
It seems that interpreting the interface report cannot be done without knowledge of the device's register layout. Therefore, I don't think the ranges can be validated/remapped automatically without involving the device driver, but there should be APIs for reading the interface report, and for requesting making specific ranges protected.
But we need to validate the interface report before accepting the device, and the device driver is only loaded after the device has been accepted.
Can we assume that only the MSI-X table and PBA ranges may be missing from the interface report, while all other non-secure regions are reported as NON-TEE ranges?
If so, we could retrieve the MSI-X guest real address details from config space and map the beginning of the BAR correctly.
Dan / Yilun — how is this handled in Intel TDX?
From what I can see, the AMD patches appear to encounter the same issue.
-aneesh
Alexey Kardashevskiy
maintainerRe: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report
Quoted 71 lines · Aneesh Kumar K.V
Quoted 62 lines · Arto Merilainen
Quoted 29 lines · Arto Merilainen
Quoted 8 lines · Aneesh Kumar K.V (Arm)
+ for (int i = 0; i <
interface_report->mmio_range_count; i++, mmio_range++) {
+
+ /*FIXME!! units in 4K size*/
+range_id=FIELD_GET(TSM_INTF_REPORT_MMIO_RANGE_ID, mmio_range->range_attributes);drivers/virt/coco/arm-cca-guest/rsi-da.c +2
-0+ + /* no secure interrupts */drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 297
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+ if (msix_tbl_bar != -1 && range_id == msix_tbl_bar) { + pr_info("Skipping misx table\n"); + continue; + } + + if (msix_pba_bar != -1 && range_id == msix_pba_bar) { + pr_info("Skipping misx pba\n"); + continue; + } +MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated. It is enough to have sufficient isolation within the BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.
Quoted 3 lines
Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?
I re-reviewed my suggestion, and what I proposed here seems wrong. However, I think there is a different more generic problem related to the MSI-X table, PBA and non-TEE ranges.
If a BAR is sparse (e.g., it has TEE pages and the MSI-X table, PBA or non-TEE areas), the TDISP interface report may contain multiple ranges with the same
range_id(/BAR id). In case a BAR contains some registers in low addresses, the MSI-X table and other registers after the MSI-X table, the interface report is expected to have two ranges for the same BAR with different "first 4k page" and "size" fields.This creates a tricky problem given that
RSI_VDEV_VALIDATE_MAPPINGrequires both theipa_baseandpa_basewhich should correspond to the same location. In above scenario, the PA of the first range would correspond to the BAR base whereas the second range would correspond to a location residing after the MSI-X table.Assuming that the report contains obfuscated (but linear) physical addresses, it would be possible to create heuristics for this case. However, the fundamental problem is that none of the "first 4k page" fields in the ranges is guaranteed to correspond to the base of any BAR:
Consider a case where the MSI-X table is in the beginning of a BAR and it is followed by a single TEE range. If the MSI-X is not locked, the "first 4k page" field will not correspond to the beginning of the BAR. If the realm naiviely reads theipa_baseusingpci_resouce_n()and correspondingpa_basefrom the interface report, the addresses won't match and the validation will fail.Quoted 6 lines
It seems that interpreting the interface report cannot be done without knowledge of the device's register layout. Therefore, I don't think the ranges can be validated/remapped automatically without involving the device driver, but there should be APIs for reading the interface report, and for requesting making specific ranges protected.
Quoted 6 lines
But we need to validate the interface report before accepting the device, and the device driver is only loaded after the device has been accepted.
Can we assume that only the MSI-X table and PBA ranges may be missing from the interface report, while all other non-secure regions are reported as NON-TEE ranges?
In case of the "some registers"+MSIX+"some more registers" BAR#X, I'd rather expect 3 ranges in the report and not just 2, for each subrange, sorted, with/without NonTEE bit set, as the PCIe spec suggests (sort of, I struggle parsing it):
===
When reporting the MMIO range for a TDI, the MMIO ranges must be reported in the logical order in which the TDI MMIO range is configured such that the first range reported corresponds to first range of pages in the TDI and so on
===
And if the number of pages in the report differs from the BAR size, then we should fail validation. Otherwise it is impossible to tell from the report's MMIO addresses what part of a BAR needs to be validated (==TEE) and if the guest device driver has to know it - then reports are just useless (which is hardly true).
Quoted 6 lines
If so, we could retrieve the MSI-X guest real address details from config space and map the beginning of the BAR correctly.
Dan / Yilun — how is this handled in Intel TDX?
From what I can see, the AMD patches appear to encounter the same issue.
I am skipping MSIX BAR because T=1 is tied to C=1 so after such validation, the BAR belongs to the guest and the hw will reject VFIO-PCI (==HV) attempts to write to MSIX BAR. Probably too straight forward though and I can try assigning it to the host (and add Cbit in those PTEs), and see how the PSP handles it (not). Thanks,
Quoted 2 lines
-aneesh
Signature · 1 line
Alexey
Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report
In reply to Aneesh Kumar K.VQuoted 2 lines · Aneesh Kumar K.V
But we need to validate the interface report before accepting the device, and the device driver is only loaded after the device has been accepted.
+1
This must work from the generic OS code.
So I'd say add a new TSM op:
int validate_pci_bar_range(struct pci_dev *pdev,
unsigned int bar_index, u64 tdisp_pa, u64 size,phys_addr_t *bar_offset_out);
TSM has broadly two options to compute bar_offset_out:
1) Require the TDISP MMIO Offset is aligned to the BAR size and use something like:
*bar_offset_out = (tdisp_pa) % pci_resource_len(pdev, bar_index);
ipa = pci_resource_start(pdev, bar_index) + *bar_offset_out;
if (size + *bar_offset_out > pci_resource_len(pdev, bar_index))
return -EINVAL;
tsm_call_to_validate(pdev, ipa, pa, size)2) Require the TSM to convert the offest'd PA to the IPA:
tsm_call_to_convert(pdev, pa, size, &ipa);
if (ipa < pci_resource_start(pdev, bar_index) ||
ipa >= pci_resource_end(pdev, bar_index) ||
(ipa + size) > pci_resource_end(pdev, bar_index))
return -EINVAL;
*bar_offset_out = ipa - pci_resource_start(pdev, bar_index);Then the generic code builds a map of what parts of the BAR are secure and what are not.
If it can't do either the TSM is unusable by Linux.
Jason
Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report
In reply to Aneesh Kumar K.VQuoted 78 lines · Aneesh Kumar K.V
Quoted 62 lines · Arto Merilainen
Quoted 29 lines · Arto Merilainen
Quoted 8 lines · Aneesh Kumar K.V (Arm)
+ for (int i = 0; i <
interface_report->mmio_range_count; i++, mmio_range++) {
+
+ /*FIXME!! units in 4K size*/
+range_id=FIELD_GET(TSM_INTF_REPORT_MMIO_RANGE_ID, mmio_range->range_attributes);drivers/virt/coco/arm-cca-guest/rsi-da.c +2
-0+ + /* no secure interrupts */drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 297
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+ if (msix_tbl_bar != -1 && range_id == msix_tbl_bar) { + pr_info("Skipping misx table\n"); + continue; + } + + if (msix_pba_bar != -1 && range_id == msix_pba_bar) { + pr_info("Skipping misx pba\n"); + continue; + } +MSI-X and PBA can be placed to a BAR that has other registers as well. While the PCIe specification recommends BAR-level isolation for MSI-X structures, it is not mandated. It is enough to have sufficient isolation within the BAR. Therefore, skipping the MSI-X and PBA BARs altogether may leave registers unintentionally mapped via unprotected IPA when they should have been mapped via protected IPA.
Quoted 3 lines
Instead of skipping the whole BAR, would it make sense to determine where the MSI-X related regions reside, and skip validation only from these regions?
I re-reviewed my suggestion, and what I proposed here seems wrong. However, I think there is a different more generic problem related to the MSI-X table, PBA and non-TEE ranges.
If a BAR is sparse (e.g., it has TEE pages and the MSI-X table, PBA or non-TEE areas), the TDISP interface report may contain multiple ranges with the same
range_id(/BAR id). In case a BAR contains some registers in low addresses, the MSI-X table and other registers after the MSI-X table, the interface report is expected to have two ranges for the same BAR with different "first 4k page" and "size" fields.This creates a tricky problem given that
RSI_VDEV_VALIDATE_MAPPINGrequires both theipa_baseandpa_basewhich should correspond to the same location. In above scenario, the PA of the first range would correspond to the BAR base whereas the second range would correspond to a location residing after the MSI-X table.Assuming that the report contains obfuscated (but linear) physical addresses, it would be possible to create heuristics for this case. However, the fundamental problem is that none of the "first 4k page" fields in the ranges is guaranteed to correspond to the base of any BAR:
Consider a case where the MSI-X table is in the beginning of a BAR and it is followed by a single TEE range. If the MSI-X is not locked, the "first 4k page" field will not correspond to the beginning of the BAR. If the realm naiviely reads theipa_baseusingpci_resouce_n()and correspondingpa_basefrom the interface report, the addresses won't match and the validation will fail.Quoted 6 lines
It seems that interpreting the interface report cannot be done without knowledge of the device's register layout. Therefore, I don't think the ranges can be validated/remapped automatically without involving the device driver, but there should be APIs for reading the interface report, and for requesting making specific ranges protected.
But we need to validate the interface report before accepting the device, and the device driver is only loaded after the device has been accepted.
Can we assume that only the MSI-X table and PBA ranges may be missing from the interface report, while all other non-secure regions are reported as NON-TEE ranges?
Quoted 6 lines
If so, we could retrieve the MSI-X guest real address details from config space and map the beginning of the BAR correctly.
Dan / Yilun — how is this handled in Intel TDX?
From what I can see, the AMD patches appear to encounter the same issue.
Same issue exists for TDX. In the near term this solidifies that the PCI/TSM core should not be assumining anything with respect to marking MMIO ranges as private, and leave that all the to low-level TSM driver.
...but then yes I expect we need to build some common infrastructure for special casing MSIX.
Re: [RFC PATCH v1 34/38] coco: guest: arm64: Validate mmio range found in the interface report
In reply to Aneesh Kumar K.V (Arm)Quoted 20 lines · Aneesh Kumar K.V (Arm)
This starts the sequence to transition the realm device to the TDISP RUN state by writing 1 to 'tsm/accept'.
+++
b/drivers/virt/coco/arm-cca-guest/rsi-da.cdrivers/virt/coco/arm-cca-guest/rsi-da.c +15
-0+int rsi_update_interface_report (struct pci_dev *pdev,bool validate) +{ +int ret; +struct resource *r; +int msix_tbl_bar, msix_pba_bar; +unsigned int range_id; +unsigned long mmio_start_phys; +unsigned long mmio_flags =0 ;/* non coherent, not limited order */ +struct pci_tdisp_mmio_range *mmio_range; +struct pci_tdisp_device_interface_report *interface_report; +struct cca_guest_dsc *dsm =to_cca_guest_dsc (pdev); +int vdev_id = (pci_domain_nr (pdev->bus) <<16 ) | +PCI_DEVID (pdev->bus->number, pdev->devfn); + +
drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 284
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+ interface_report = (struct pci_tdisp_device_interface_report *)dsm->interface_report; + mmio_range = (struct pci_tdisp_mmio_range *)(interface_report +1 ); + + + msix_tbl_bar =get_msix_bar (pdev,PCI_MSIX_TABLE ); + msix_pba_bar =get_msix_bar (pdev,PCI_MSIX_PBA ); + +for (int i =0 ; i < interface_report->mmio_range_count; i++, mmio_range++) { + +/*FIXME!! units in 4K size*/
I guess you intend to fix this?
drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 294
Quoted 8 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +8
-0+/* no secure interrupts */ +if (msix_tbl_bar !=-1 && range_id == msix_tbl_bar) { +pr_info ("Skipping misx table\n" ); +continue ; + } + +if (msix_pba_bar !=-1 && range_id == msix_pba_bar) { +pr_info ("Skipping misx pba\n" );
s/misx/MSI-X/ (twice)
s/pba/PBA/
Aneesh Kumar K.V (Arm)
maintainerRFC PATCH 35/38 coco: guest: arm64: Add Realm device start and stop support
In reply to Aneesh Kumar K.V (Arm)Writing 1 to 'tsm/acceept' will initiate the TDISP RUN sequence.
--- arch/arm64/include/asm/rsi_cmds.h | 19 +++++++ arch/arm64/include/asm/rsi_smc.h | 2 + drivers/virt/coco/arm-cca-guest/arm-cca.c | 15 ++++++ drivers/virt/coco/arm-cca-guest/rsi-da.c | 60 +++++++++++++++++++++++ drivers/virt/coco/arm-cca-guest/rsi-da.h | 2 +- 5 files changed, 97 insertions(+), 1 deletion(-)
arch/arm64/include/asm/rsi_cmds.h +19 -0
@@ -246,4 +246,23 @@ rsi_rdev_validate_mapping(unsigned long vdev_id, unsigned long inst_id,return -EINVAL ; return 0; } + +static inline unsigned long __rsi_rdev_start (unsigned long vdev_id,unsigned long inst_id) +{ +struct arm_smccc_res res; + +arm_smccc_1_1_invoke (SMC_RSI_RDEV_START , vdev_id, inst_id, &res); + +return res.a0; +} + +static inline unsigned long __rsi_rdev_stop (unsigned long vdev_id,unsigned long inst_id) +{ +struct arm_smccc_res res; + +arm_smccc_1_1_invoke (SMC_RSI_RDEV_STOP , vdev_id, inst_id, &res); + +return res.a0; +} +#endif /* __ASM_RSI_CMDS_H */
arch/arm64/include/asm/rsi_smc.h +2 -0
@@ -203,6 +203,8 @@ struct rsi_host_call {#define SMC_RSI_RDEV_GET_INTERFACE_REPORT SMC_RSI_FID(0x1a6)#define SMC_RSI_RDEV_LOCK SMC_RSI_FID(0x1a9) +#define SMC_RSI_RDEV_START SMC_RSI_FID(0x1aa) +#define SMC_RSI_RDEV_STOP SMC_RSI_FID(0x1ab)#define RSI_DEV_MEM_COHERENT BIT(0)#define RSI_DEV_MEM_LIMITED_ORDER BIT(1)
drivers/virt/coco/arm-cca-guest/arm-cca.c +15 -0
@@ -250,6 +250,8 @@ static void cca_tsm_unlock(struct pci_dev *pdev)/* invalidate dev mapping based on interface report */ rsi_update_interface_report (pdev, false); +rsi_device_stop (pdev); + ret =rhi_da_vdev_set_tdi_state (vdev_id,RHI_DA_TDI_CONFIG_UNLOCKED );if (ret) {pci_err (pdev,"failed to TSM unbind the device (%ld)\n" , ret);
@@ -257,11 +259,24 @@ static void cca_tsm_unlock(struct pci_dev *pdev) } } +static int cca_tsm_accept (struct pci_dev *pdev) +{ +int ret; + + ret =rsi_device_start (pdev); +if (ret) { +pci_err (pdev,"failed to transition the device to run state (%d)\n" , ret); +return -EIO ; + } +return 0 ; +} +static const struct pci_tsm_ops cca_pci_ops = { .probe = cca_tsm_pci_probe, .remove = cca_tsm_pci_remove, .lock = cca_tsm_lock, .unlock = cca_tsm_unlock, + .accept = cca_tsm_accept, };static void cca_tsm_unregister (void *tsm)
drivers/virt/coco/arm-cca-guest/rsi-da.c +60 -0
77 lines not on this page · raw message
drivers/virt/coco/arm-cca-guest/rsi-da.h +1 -1
@@ -54,5 +54,5 @@ static inline struct cca_guest_dsc *to_cca_guest_dsc(struct pci_dev *pdev)int rsi_update_interface_report (struct pci_dev *pdev,bool validate);int rsi_device_lock (struct pci_dev *pdev);int rsi_device_start (struct pci_dev *pdev); - +int rsi_device_stop (struct pci_dev *pdev);#endif
Signature · 1 line
2.43.0
Re: [RFC PATCH v1 35/38] coco: guest: arm64: Add Realm device start and stop support
Quoted 3 lines · On Mon, 28 Jul 2025 19:22:12 +0530 "Aneesh Kumar K.V (Arm)"…
Writing 1 to 'tsm/acceept' will initiate the TDISP RUN sequence.
Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>
A few more trivial things on this first read through.
Jonathan
drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_start(struct pci_dev *pdev) · line 401
Quoted 11 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +6
-0+int rsi_device_stop (struct pci_dev *pdev) +{ +int ret; +struct cca_guest_dsc *dsm =to_cca_guest_dsc (pdev); +int vdev_id = (pci_domain_nr (pdev->bus) <<16 ) | +PCI_DEVID (pdev->bus->number, pdev->devfn);
Feels like this occurs so often we should add a helper. Can't be completely generic as pci_domain_nr can have more bits than I guess we are assuming here, but we can have something for use in the rsi code.
Quoted 13 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +6
-0+ + ret =rsi_rdev_stop (pdev, vdev_id, dsm->instance_id); +if (ret !=RSI_SUCCESS ) { +pci_err (pdev,"failed to stop the device (%u)\n" , ret); +return -EIO ; + }return 0 ; }drivers/virt/coco/arm-cca-guest/rsi-da.h +0
-0
drivers/virt/coco/arm-cca-guest/rsi-da.h static inline struct cca_guest_dsc *to_cca_guest_dsc(struct pci_dev *pdev) · line 57
Quoted 4 lines
drivers/virt/coco/arm-cca-guest/rsi-da.h +0
-1@@ -54,5 +54,5 @@ static inline struct cca_guest_dsc *to_cca_guest_dsc(struct pci_dev *pdev)int rsi_update_interface_report (struct pci_dev *pdev,bool validate);int rsi_device_lock (struct pci_dev *pdev);int rsi_device_start (struct pci_dev *pdev); -
Tidy this up.
Quoted 2 lines
drivers/virt/coco/arm-cca-guest/rsi-da.h +1
-0+int rsi_device_stop (struct pci_dev *pdev);#endif
Re: [RFC PATCH v1 35/38] coco: guest: arm64: Add Realm device start and stop support
In reply to Aneesh Kumar K.V (Arm)Quoted 1 line · Aneesh Kumar K.V (Arm)
Writing 1 to 'tsm/acceept' will initiate the TDISP RUN sequence.
s/acceept/accept/
Aneesh Kumar K.V (Arm)
maintainerRFC PATCH 36/38 KVM: arm64: CCA: enable DA in realm create parameters
In reply to Aneesh Kumar K.V (Arm)Now that we have all the required steps for DA in-place, enable DA while creating ralm.
--- arch/arm64/include/asm/rmi_smc.h | 1 + arch/arm64/kvm/rme.c | 2 ++ 2 files changed, 3 insertions(+)
arch/arm64/include/asm/rmi_smc.h +1 -0
@@ -112,6 +112,7 @@ enum rmi_ripas {#define RMI_REALM_PARAM_FLAG_LPA2 BIT(0)#define RMI_REALM_PARAM_FLAG_SVE BIT(1)#define RMI_REALM_PARAM_FLAG_PMU BIT(2) +#define RMI_REALM_PARAM_FLAG_DA BIT(3)/* * Note many of these fields are smaller than u64 but all fields have u64
arch/arm64/kvm/rme.c +2 -0
@@ -695,6 +695,8 @@ static int realm_create_rd(struct kvm *kvm)if (r) goto out_undelegate_tables; +/* For now default enable DA */ + params->flags =RMI_REALM_PARAM_FLAG_DA ; params_phys =virt_to_phys (params);if (rmi_realm_create (rd_phys, params_phys)) {
Signature · 1 line
2.43.0
Re: [RFC PATCH v1 36/38] KVM: arm64: CCA: enable DA in realm create parameters
Quoted 2 lines · Aneesh Kumar K.V (Arm)
Now that we have all the required steps for DA in-place, enable DA while creating ralm.
s/ralm/realm/ ?
Aneesh Kumar K.V (Arm)
maintainerRFC PATCH 37/38 coco: guest: arm64: Add support for fetching device measurements
In reply to Aneesh Kumar K.V (Arm)Fetch device measurements using RSI_RDEV_GET_MEASUREMENTS.
--- arch/arm64/include/asm/rsi_cmds.h | 11 +++++++ arch/arm64/include/asm/rsi_smc.h | 16 ++++++++++ drivers/virt/coco/arm-cca-guest/rsi-da.c | 39 ++++++++++++++++++++++++ drivers/virt/coco/arm-cca-guest/rsi-da.h | 2 ++ 4 files changed, 68 insertions(+)
arch/arm64/include/asm/rsi_cmds.h +11 -0
@@ -265,4 +265,15 @@ static inline unsigned long __rsi_rdev_stop(unsigned long vdev_id, unsigned long return res.a0; } +static inline unsigned long __rsi_rdev_get_measurements (unsigned long vdev_id, +unsigned long inst_id, +phys_addr_t meas) +{ +struct arm_smccc_res res; + +arm_smccc_1_1_invoke (SMC_RSI_RDEV_GET_MEASUREMENTS , vdev_id, inst_id, meas, &res); + +return res.a0; +} +#endif /* __ASM_RSI_CMDS_H */
arch/arm64/include/asm/rsi_smc.h +16 -0
@@ -202,6 +202,22 @@ struct rsi_host_call {#define SMC_RSI_RDEV_GET_INTERFACE_REPORT SMC_RSI_FID(0x1a6) +#define RSI_DEV_MEASURE_ALL BIT(0) +#define RSI_DEV_MEASURE_SIGNED BIT(1) +#define RSI_DEV_MEASURE_RAW BIT(2) + +struct rsi_device_measurements_params { +union { +struct { +u64 flags; +u8 indices[32 ]; +u8 nounce[32 ]; + }; +u8 padding[0x100 ]; + }; +}; + +#define SMC_RSI_RDEV_GET_MEASUREMENTS SMC_RSI_FID(0x1a7)#define SMC_RSI_RDEV_LOCK SMC_RSI_FID(0x1a9)#define SMC_RSI_RDEV_START SMC_RSI_FID(0x1aa)#define SMC_RSI_RDEV_STOP SMC_RSI_FID(0x1ab)
drivers/virt/coco/arm-cca-guest/rsi-da.c +39 -0
@@ -166,10 +166,31 @@ static long rhi_get_report(int vdev_id, int da_object_type, void **report, int *return ret; } +static inline unsigned long +rsi_rdev_get_measurements (struct pci_dev *pdev,unsigned long vdev_id, +unsigned long inst_id,phys_addr_t meas) +{ +unsigned long ret; + + ret =__rsi_rdev_get_measurements (vdev_id, inst_id, meas); +if (ret !=RSI_SUCCESS ) +return ret; + +do { + ret =rsi_rdev_continue (vdev_id, inst_id); + }while (ret ==RSI_INCOMPLETE ); +if (ret !=RSI_SUCCESS ) { +pci_err (pdev,"failed to communicate with the device (%lu)\n" , ret); +return ret; + } +return RSI_SUCCESS ; +} +int rsi_device_lock (struct pci_dev *pdev) {unsigned long ret;unsigned long tdisp_version; +struct rsi_device_measurements_params *rsi_dev_meas;struct cca_guest_dsc *dsm =to_cca_guest_dsc (pdev);int vdev_id = (pci_domain_nr (pdev->bus) <<16 ) |PCI_DEVID (pdev->bus->number, pdev->devfn);
@@ -198,6 +219,17 @@ int rsi_device_lock(struct pci_dev *pdev)return -EOPNOTSUPP ; } + rsi_dev_meas = (struct rsi_device_measurements_params *)__get_free_page (GFP_KERNEL ); + rsi_dev_meas->flags =RSI_DEV_MEASURE_ALL ; + ret =rsi_rdev_get_measurements (pdev, vdev_id, dsm->instance_id, +virt_to_phys (rsi_dev_meas)); + +free_page ((unsigned long )rsi_dev_meas); +if (ret !=RSI_SUCCESS ) { +pci_err (pdev,"failed to get device measurement (%lu)\n" , ret); +return -EIO ; + } +/* Now make a host call to copy the interface report to guest. */ ret =rhi_get_report (vdev_id,RHI_DA_OBJECT_INTERFACE_REPORT , &dsm->interface_report, &dsm->interface_report_size);
@@ -213,6 +245,13 @@ int rsi_device_lock(struct pci_dev *pdev)return -EIO ; } + ret =rhi_get_report (vdev_id,RHI_DA_OBJECT_MEASUREMENT , + &dsm->measurements, &dsm->measurements_size); +if (ret) { +pci_err (pdev,"failed to get device certificate from the host (%lu)\n" , ret); +return -EIO ; + } +return ret; } static inline unsigned long rsi_rdev_start(struct pci_dev *pdev,
drivers/virt/coco/arm-cca-guest/rsi-da.h +2 -0
@@ -40,6 +40,8 @@ struct cca_guest_dsc {int interface_report_size;void *certificate;int certificate_size; +void *measurements; +int measurements_size; };static inline struct cca_guest_dsc *to_cca_guest_dsc (struct pci_dev *pdev)
Signature · 1 line
2.43.0
Re: [RFC PATCH v1 37/38] coco: guest: arm64: Add support for fetching device measurements
Quoted 3 lines · On Mon, 28 Jul 2025 19:22:14 +0530 "Aneesh Kumar K.V (Arm)"…
Fetch device measurements using
RSI_RDEV_GET_MEASUREMENTS.Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>
One completely trivial comment.
J
Quoted 7 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +0
-0@@ -213,6 +245,13 @@ int rsi_device_lock(struct pci_dev *pdev)return -EIO ;
drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 255
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +7
-0} + ret =rhi_get_report (vdev_id,RHI_DA_OBJECT_MEASUREMENT , + &dsm->measurements, &dsm->measurements_size); +if (ret) { +pci_err (pdev,"failed to get device certificate from the host (%lu)\n" , ret); +return -EIO ; + } +return ret;
return 0; Always good to make it explicit when it can't take any other values. Looks like that belong sin an earlier patch though based on this snippet.
Quoted 1 line
drivers/virt/coco/arm-cca-guest/rsi-da.c +0
-0}
Re: [RFC PATCH v1 37/38] coco: guest: arm64: Add support for fetching device measurements
In reply to Aneesh Kumar K.V (Arm)arch/arm64/include/asm/rsi_smc.h struct rsi_host_call { · line 214
Quoted 10 lines · Aneesh Kumar K.V (Arm)
Fetch device measurements using
RSI_RDEV_GET_MEASUREMENTS.+++
b/arch/arm64/include/asm/rsi_smc.harch/arm64/include/asm/rsi_smc.h +6
-0+struct rsi_device_measurements_params { + union { + struct { + u64 flags; + u8 indices[32]; + u8 nounce[32];
s/nounce/nonce/ ?
Aneesh Kumar K.V (Arm)
maintainerRFC PATCH 38/38 coco: guest: arm64: Add support for fetching device info
In reply to Aneesh Kumar K.V (Arm)RSI_RDEV_GET_INFO returns different digest hash values, which can be compared with host cached values to ensure the host didn't tamper with the cached data.
--- arch/arm64/include/asm/rsi_cmds.h | 12 +++ arch/arm64/include/asm/rsi_smc.h | 24 +++++ drivers/virt/coco/arm-cca-guest/Kconfig | 2 + drivers/virt/coco/arm-cca-guest/rsi-da.c | 128 +++++++++++++++++++++++ drivers/virt/coco/arm-cca-guest/rsi-da.h | 13 +++ 5 files changed, 179 insertions(+)
arch/arm64/include/asm/rsi_cmds.h +12 -0
@@ -276,4 +276,16 @@ static inline unsigned long __rsi_rdev_get_measurements(unsigned long vdev_id, return res.a0; } +static inline unsigned long rsi_rdev_get_info (unsigned long vdev_id, +unsigned long inst_id, +unsigned long digest_phys) +{ +struct arm_smccc_res res; + +arm_smccc_1_1_invoke (SMC_RSI_RDEV_GET_INFO , + vdev_id, inst_id, digest_phys, &res); + +return res.a0; +} +#endif /* __ASM_RSI_CMDS_H */
arch/arm64/include/asm/rsi_smc.h +24 -0
@@ -125,6 +125,9 @@ #ifndef __ASSEMBLY__ +#define RSI_HASH_SHA_256 0 +#define RSI_HASH_SHA_512 1 + struct realm_config { union { struct {
@@ -200,6 +203,27 @@ struct rsi_host_call {#define SMC_RSI_RDEV_GET_INSTANCE_ID SMC_RSI_FID(0x19c)#define SMC_RSI_RDEV_CONTINUE SMC_RSI_FID(0x1a4) +struct rsi_device_info { +union { +struct { +u64 flags; +u64 attest_type; +u64 cert_id; +u8 hash_algo; + }; +u8 padding[0x40 ]; + }; +union {/* 0x40 */ +struct { +u8 cert_digest[0x40 ]; +u8 meas_digest[0x40 ]; +u8 report_digest[0x40 ]; + }; +u8 padding2[0x200 -0x40 ]; + }; +}; + +#define SMC_RSI_RDEV_GET_INFO SMC_RSI_FID(0x1a5)#define SMC_RSI_RDEV_GET_INTERFACE_REPORT SMC_RSI_FID(0x1a6)#define RSI_DEV_MEASURE_ALL BIT(0)
drivers/virt/coco/arm-cca-guest/Kconfig +2 -0
@@ -5,6 +5,8 @@ config ARM_CCA_GUEST tristate "Arm CCA Guest driver" depends on ARM64 depends on PCI_TSM + select CRYPTO_SHA256 + select CRYPTO_SHA512 select TSM_REPORTS select TSM help
drivers/virt/coco/arm-cca-guest/rsi-da.c +128 -0
153 lines not on this page · raw message
drivers/virt/coco/arm-cca-guest/rsi-da.h +13 -0
@@ -10,6 +10,7 @@#include <linux/pci-tsm.h> #include <asm/rsi_smc.h> #include <asm/rhi.h> +#include <crypto/sha2.h> struct pci_tdisp_device_interface_report {u16 interface_info;
@@ -33,6 +34,17 @@ struct pci_tdisp_mmio_range {#define TSM_INTF_REPORT_MMIO_RESERVED GENMASK(15, 4)#define TSM_INTF_REPORT_MMIO_RANGE_ID GENMASK(31, 16) +struct dsm_device_info { +u64 flags; +u64 attest_type; +u64 cert_id; +u64 hash_algo; +u8 cert_digest[SHA512_DIGEST_SIZE ]; +u8 meas_digest[SHA512_DIGEST_SIZE ]; +u8 report_digest[SHA512_DIGEST_SIZE ]; +}; + +struct cca_guest_dsc {struct pci_tsm_pf0 pci;unsigned long instance_id;
@@ -42,6 +54,7 @@ struct cca_guest_dsc {int certificate_size;void *measurements;int measurements_size; +struct dsm_device_info dev_info; };static inline struct cca_guest_dsc *to_cca_guest_dsc (struct pci_dev *pdev)
Signature · 1 line
2.43.0
Re: [RFC PATCH v1 38/38] coco: guest: arm64: Add support for fetching device info
Quoted 5 lines · On Mon, 28 Jul 2025 19:22:15 +0530 "Aneesh Kumar K.V (Arm)"…
RSI_RDEV_GET_INFOreturns different digest hash values, which can be compared with host cached values to ensure the host didn't tamper with the cached data.Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>
Hi Aneesh
A few comments on this one
Jonathan
Quoted 41 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +26
-0@@ -6,6 +6,7 @@#include <linux/pci.h> #include <linux/mem_encrypt.h> #include <asm/rsi_cmds.h> +#include <crypto/hash.h> #include "rsi-da.h" @@ -186,11 +187,102 @@ rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id,return RSI_SUCCESS ; } +static int verify_digests (struct cca_guest_dsc *dsm) +{ +int i; +int ret; +u8 digest[SHA512_DIGEST_SIZE ]; +int sdesc_size; +size_t digest_size; +char *hash_alg_name; +struct shash_desc *shash; +struct crypto_shash *alg; +struct pci_dev *pdev = dsm->pci.tsm.pdev; +struct { +uint8_t *report; +size_t size; +uint8_t *digest; + } reports[] = { + { + dsm->interface_report, + dsm->interface_report_size, + dsm->dev_info.report_digest + }, + { + dsm->certificate, + dsm->certificate_size, + dsm->dev_info.cert_digest
drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 224
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+ }, + { + dsm->measurements, + dsm->measurements_size, + dsm->dev_info.meas_digest + } + }; + + +if (dsm->dev_info.hash_algo ==RSI_HASH_SHA_256 ) {
This to me smells like a place that will need switch sooner or later, maybe just do it now.
drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 232
Quoted 8 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +8
-0+ hash_alg_name ="sha256" ; + digest_size =SHA256_DIGEST_SIZE ; + }else if (dsm->dev_info.hash_algo ==RSI_HASH_SHA_512 ) { + hash_alg_name ="sha512" ; + digest_size =SHA512_DIGEST_SIZE ; + }else { +pci_err (pdev,"unknown realm hash algorithm!\n" ); + ret = -EINVAL ;
return -EINVAL;
drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 236
Quoted 4 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +4
-0+ goto err_out; + } + + alg =crypto_alloc_shash (hash_alg_name,0 ,0 );
As below - I'd spin a DEFINE_FREE() for this to simplify error paths in here and remove the labels that had me confused briefly.
drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 242
Quoted 6 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +6
-0+if (IS_ERR (alg)) { +pci_err (pdev,"cannot allocate %s\n" , hash_alg_name); +return PTR_ERR (alg); + } + + sdesc_size =sizeof (struct shash_desc ) +crypto_shash_descsize (alg);
Not common in crypto so perhaps just leave this as you have it.
sdesc_size = struct_size(struct shash_dec, __ctx, crypto_shash_desc_size(alg));
is more informative on what is going on here.
drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 253
Quoted 11 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +11
-0+ shash =kzalloc (sdesc_size,GFP_KERNEL ); +if (!shash) { +pci_err (pdev,"cannot allocate sdesc\n" ); + ret = -ENOMEM ; + goto err_free_shash; + } + shash->tfm = alg; + +for (i =0 ; i <ARRAY_SIZE (reports); i++) { + ret =crypto_shash_digest (shash, reports[i].report, + reports[i].size, digest);
To me a bit marginal on whether this loop and the structures above are beneficial over straight line code.
Quoted 6 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +6
-0+if (ret) { +pci_err (pdev,"failed to compute digest, %d\n" , ret); + goto err_free_sdesc; + } + +if (memcmp (reports[i].digest, digest, digest_size)) {
drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 269
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+pci_err (pdev,"invalid digest\n" ); + ret = -EINVAL ; + goto err_free_sdesc; + } + } + +kfree (shash); +crypto_free_shash (alg); + +pci_info (pdev,"Successfully verified the digests\n" );
debug.
drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 272
Quoted 3 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +3
-0+return 0 ; + +err_free_sdesc:
I'd tweak these labels if you keep them. This isn't freeing the sdesc.
drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 273
Quoted 1 line
drivers/virt/coco/arm-cca-guest/rsi-da.c +1
-0+kfree (shash);
Looks perfect for some __free() magic dust.
drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 275
Quoted 2 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +2
-0+err_free_shash: +crypto_free_shash (alg);
DEFINE_FREE() needed for this but looks pretty uncontroversial.
drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 276
Quoted 1 line
drivers/virt/coco/arm-cca-guest/rsi-da.c +1
-0+err_out:
As below. I'd not do this.
Quoted 13 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +4
-0+return ret; +} +int rsi_device_lock (struct pci_dev *pdev) {unsigned long ret;unsigned long tdisp_version;struct rsi_device_measurements_params *rsi_dev_meas; +struct rsi_device_info *dev_info;struct cca_guest_dsc *dsm =to_cca_guest_dsc (pdev);int vdev_id = (pci_domain_nr (pdev->bus) <<16 ) |PCI_DEVID (pdev->bus->number, pdev->devfn);
drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 348
Quoted 5 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +2
-0@@ -252,8 +344,44 @@ int rsi_device_lock(struct pci_dev *pdev)return -EIO ; } +/* RMM expects sizeof(dev_info) (512 bytes) aligned address */ + dev_info =kmalloc (sizeof (*dev_info),GFP_KERNEL );
Use a __free(kfree) here (and direct returns on errors) given it's freed in all paths and we don't care if it is freed before or after verifying the digests.
I'm being slow today, but what in that enforces the alignment? I guess it's that the structure happens to be big enough that it happens naturally?
I'd allocate max(512, sizeof(*dev_info)) to make it explicitly the case.
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+if (!dev_info) { + ret = -ENOMEM ; + goto err_out; + } + + ret =rsi_rdev_get_info (vdev_id, dsm->instance_id,virt_to_phys (dev_info)); +if (ret !=RSI_SUCCESS ) { +pci_err (pdev,"failed to get device digests (%lu)\n" , ret); + ret = -EIO ; +kfree (dev_info);
drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 368
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+ goto err_out; + } + + dsm->dev_info.attest_type = dev_info->attest_type; + dsm->dev_info.cert_id = dev_info->cert_id; + dsm->dev_info.hash_algo = dev_info->hash_algo; +memcpy (dsm->dev_info.cert_digest, dev_info->cert_digest,SHA512_DIGEST_SIZE ); +memcpy (dsm->dev_info.meas_digest, dev_info->meas_digest,SHA512_DIGEST_SIZE ); +memcpy (dsm->dev_info.report_digest, dev_info->report_digest,SHA512_DIGEST_SIZE ); +
Can't you memcpy the whole thing in one go?
drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 381
Quoted 13 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +13
-0+kfree (dev_info); +/* +* Verify that the digests of the provided reports match with the +* digests from RMM +*/ + ret =verify_digests (dsm); +if (ret) { +pci_err (pdev,"device digest validation failed (%ld)\n" , ret); +return ret; + } + +return 0 ; +err_out:
I'll always grumble about these. To me it always makes the code less readable. Some others disagree though ;(
drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 384
Quoted 3 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +1
-0return ret; } +
Looks like this should have been in an earlier patch.
Quoted 19 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +0
-0static inline unsigned long rsi_rdev_start (struct pci_dev *pdev,unsigned long vdev_id,unsigned long inst_id) {drivers/virt/coco/arm-cca-guest/rsi-da.h +1
-0@@ -10,6 +10,7 @@#include <linux/pci-tsm.h> #include <asm/rsi_smc.h> #include <asm/rhi.h> +#include <crypto/sha2.h> struct pci_tdisp_device_interface_report {u16 interface_info;@@ -33,6 +34,17 @@ struct pci_tdisp_mmio_range {#define TSM_INTF_REPORT_MMIO_RESERVED GENMASK(15, 4)#define TSM_INTF_REPORT_MMIO_RANGE_ID GENMASK(31, 16)
drivers/virt/coco/arm-cca-guest/rsi-da.h struct pci_tdisp_mmio_range { · line 46
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.h +10
-0+struct dsm_device_info { +u64 flags; +u64 attest_type; +u64 cert_id; +u64 hash_algo; +u8 cert_digest[SHA512_DIGEST_SIZE ]; +u8 meas_digest[SHA512_DIGEST_SIZE ]; +u8 report_digest[SHA512_DIGEST_SIZE ]; +}; +
One probably enough.
Quoted 12 lines
drivers/virt/coco/arm-cca-guest/rsi-da.h +2
-0+struct cca_guest_dsc {struct pci_tsm_pf0 pci;unsigned long instance_id;@@ -42,6 +54,7 @@ struct cca_guest_dsc {int certificate_size;void *measurements;int measurements_size; +struct dsm_device_info dev_info; };static inline struct cca_guest_dsc *to_cca_guest_dsc (struct pci_dev *pdev)
Aneesh Kumar K.V
maintainerRe: [RFC PATCH v1 38/38] coco: guest: arm64: Add support for fetching device info
Quoted 3 lines · Jonathan Cameron
On Mon, 28 Jul 2025 19:22:15 +0530
"Aneesh Kumar K.V (Arm)" <[email protected]> wrote:
...
Quoted 22 lines
Quoted 13 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +4
-0+return ret; +} +int rsi_device_lock (struct pci_dev *pdev) {unsigned long ret;unsigned long tdisp_version;struct rsi_device_measurements_params *rsi_dev_meas; +struct rsi_device_info *dev_info;struct cca_guest_dsc *dsm =to_cca_guest_dsc (pdev);int vdev_id = (pci_domain_nr (pdev->bus) <<16 ) |PCI_DEVID (pdev->bus->number, pdev->devfn);drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 348
Quoted 5 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +2
-0@@ -252,8 +344,44 @@ int rsi_device_lock(struct pci_dev *pdev)return -EIO ; } +/* RMM expects sizeof(dev_info) (512 bytes) aligned address */ + dev_info =kmalloc (sizeof (*dev_info),GFP_KERNEL );Use a
__free(kfree)here (and direct returns on errors) given it's freed in all paths and we don't care if it is freed before or after verifying the digests.
Quoted 5 lines
I'm being slow today, but what in that enforces the alignment? I guess it's that the structure happens to be big enough that it happens naturally?
I'd allocate
max(512, sizeof(*dev_info))to make it explicitly the case.
yes, struct rsi_device_info is 512 bytes.
Quoted 15 lines
Quoted 15 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +14
-0+if (!dev_info) { + ret = -ENOMEM ; + goto err_out; + } + + ret =rsi_rdev_get_info (vdev_id, dsm->instance_id,virt_to_phys (dev_info)); +if (ret !=RSI_SUCCESS ) { +pci_err (pdev,"failed to get device digests (%lu)\n" , ret); + ret = -EIO ; +kfree (dev_info); + goto err_out; + } + + dsm->dev_info.attest_type = dev_info->attest_type;
Quoted 9 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 368
Quoted 6 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +6
-0+ dsm->dev_info.cert_id = dev_info->cert_id; + dsm->dev_info.hash_algo = dev_info->hash_algo; +memcpy (dsm->dev_info.cert_digest, dev_info->cert_digest,SHA512_DIGEST_SIZE ); +memcpy (dsm->dev_info.meas_digest, dev_info->meas_digest,SHA512_DIGEST_SIZE ); +memcpy (dsm->dev_info.report_digest, dev_info->report_digest,SHA512_DIGEST_SIZE ); +Can't you memcpy the whole thing in one go?
yes. But won't that be confusing? Is there a difference? Also struct dsm_device_info is not same as struct rsi_device_info. We don't need to keep all that padding in dsm_device_info.
Quoted 65 lines
Quoted 4 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +3
-0+kfree (dev_info); +/* +* Verify that the digests of the provided reports match with the drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 381
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+* digests from RMM +*/ + ret =verify_digests (dsm); +if (ret) { +pci_err (pdev,"device digest validation failed (%ld)\n" , ret); +return ret; + } + +return 0 ; +err_out:I'll always grumble about these. To me it always makes the code less readable. Some others disagree though ;(
drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 384
Quoted 3 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +1
-0return ret; } +Looks like this should have been in an earlier patch.
Quoted 19 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +0
-0static inline unsigned long rsi_rdev_start (struct pci_dev *pdev,unsigned long vdev_id,unsigned long inst_id) {drivers/virt/coco/arm-cca-guest/rsi-da.h +1
-0@@ -10,6 +10,7 @@#include <linux/pci-tsm.h> #include <asm/rsi_smc.h> #include <asm/rhi.h> +#include <crypto/sha2.h> struct pci_tdisp_device_interface_report {u16 interface_info;@@ -33,6 +34,17 @@ struct pci_tdisp_mmio_range {#define TSM_INTF_REPORT_MMIO_RESERVED GENMASK(15, 4)#define TSM_INTF_REPORT_MMIO_RANGE_ID GENMASK(31, 16)drivers/virt/coco/arm-cca-guest/rsi-da.h struct pci_tdisp_mmio_range { · line 46
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.h +10
-0+struct dsm_device_info { +u64 flags; +u64 attest_type; +u64 cert_id; +u64 hash_algo; +u8 cert_digest[SHA512_DIGEST_SIZE ]; +u8 meas_digest[SHA512_DIGEST_SIZE ]; +u8 report_digest[SHA512_DIGEST_SIZE ]; +}; +One probably enough.
Quoted 11 lines
drivers/virt/coco/arm-cca-guest/rsi-da.h +2
-0+struct cca_guest_dsc {struct pci_tsm_pf0 pci;unsigned long instance_id;@@ -42,6 +54,7 @@ struct cca_guest_dsc {int certificate_size;void *measurements;int measurements_size; +struct dsm_device_info dev_info; };
Quoted 1 line
drivers/virt/coco/arm-cca-guest/rsi-da.h struct cca_guest_dsc { · line 60
Quoted 1 line
drivers/virt/coco/arm-cca-guest/rsi-da.h +0
-0static inline struct cca_guest_dsc *to_cca_guest_dsc (struct pci_dev *pdev)
-aneesh
Re: [RFC PATCH v1 38/38] coco: guest: arm64: Add support for fetching device info
Quoted 26 lines
Quoted 16 lines
Quoted 16 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +14
-0+if (!dev_info) { + ret = -ENOMEM ; + goto err_out; + } + + ret =rsi_rdev_get_info (vdev_id, dsm->instance_id,virt_to_phys (dev_info)); +if (ret !=RSI_SUCCESS ) { +pci_err (pdev,"failed to get device digests (%lu)\n" , ret); + ret = -EIO ; +kfree (dev_info); + goto err_out; + } + + dsm->dev_info.attest_type = dev_info->attest_type;Quoted 9 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c int rsi_device_lock(struct pci_dev *pdev) · line 368
Quoted 6 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +6
-0+ dsm->dev_info.cert_id = dev_info->cert_id; + dsm->dev_info.hash_algo = dev_info->hash_algo; +memcpy (dsm->dev_info.cert_digest, dev_info->cert_digest,SHA512_DIGEST_SIZE ); +memcpy (dsm->dev_info.meas_digest, dev_info->meas_digest,SHA512_DIGEST_SIZE ); +memcpy (dsm->dev_info.report_digest, dev_info->report_digest,SHA512_DIGEST_SIZE ); +Can't you memcpy the whole thing in one go?
Quoted 3 lines
yes. But won't that be confusing? Is there a difference? Also struct
dsm_device_infois not same as structrsi_device_info. We don't need to keep all that padding indsm_device_info.
Ah. I misread and thought they were the same structure. No problem copying only relevant fields then!
Jonathan
Re: [RFC PATCH v1 38/38] coco: guest: arm64: Add support for fetching device info
In reply to Aneesh Kumar K.V (Arm)drivers/virt/coco/arm-cca-guest/Kconfig config ARM_CCA_GUEST · line 11
Quoted 8 lines · Aneesh Kumar K.V (Arm)
drivers/virt/coco/arm-cca-guest/Kconfig +2
-0@@ -5,6 +5,8 @@ config ARM_CCA_GUEST tristate "Arm CCA Guest driver" depends on ARM64 depends on PCI_TSM + select CRYPTO_SHA256 + select CRYPTO_SHA512 select TSM_REPORTS select TSM
CRYPTO_LIB_SHA256 and CRYPTO_LIB_SHA512
Quoted 24 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +24
-0+if (dsm->dev_info.hash_algo ==RSI_HASH_SHA_256 ) { + hash_alg_name ="sha256" ; + digest_size =SHA256_DIGEST_SIZE ; + }else if (dsm->dev_info.hash_algo ==RSI_HASH_SHA_512 ) { + hash_alg_name ="sha512" ; + digest_size =SHA512_DIGEST_SIZE ; + }else { +pci_err (pdev,"unknown realm hash algorithm!\n" ); + ret = -EINVAL ; + goto err_out; + } + + alg =crypto_alloc_shash (hash_alg_name,0 ,0 ); +if (IS_ERR (alg)) { +pci_err (pdev,"cannot allocate %s\n" , hash_alg_name); +return PTR_ERR (alg); + } + + sdesc_size =sizeof (struct shash_desc ) +crypto_shash_descsize (alg); + shash =kzalloc (sdesc_size,GFP_KERNEL ); +if (!shash) { +pci_err (pdev,"cannot allocate sdesc\n" ); + ret = -ENOMEM ; + goto err_free_shash;
drivers/virt/coco/arm-cca-guest/rsi-da.c rsi_rdev_get_measurements(struct pci_dev *pdev, unsigned long vdev_id, · line 257
Quoted 10 lines
drivers/virt/coco/arm-cca-guest/rsi-da.c +10
-0+ } + shash->tfm = alg; + +for (i =0 ; i <ARRAY_SIZE (reports); i++) { + ret =crypto_shash_digest (shash, reports[i].report, + reports[i].size, digest); +if (ret) { +pci_err (pdev,"failed to compute digest, %d\n" , ret); + goto err_free_sdesc; + }
Use sha256() and sha512().
- Eric
Quoted 3 lines · Aneesh Kumar K.V (Arm)
This patch series implements support for Device Assignment in the ARM CCA architecture. The code changes are based on Alp12 specification published here [1].
Robin and I were talking about CCA and DMA here:
https://lore.kernel.org/r/[email protected]
What do you think about pulling some of this out and trying to independently push a series getting the DMA API layers ready for device assignment?
I think there will be some discussion on these points, it would be good to get started.
Jason
Quoted 9 lines · Aneesh Kumar K.V (Arm)
This patch series implements support for Device Assignment in the ARM CCA architecture. The code changes are based on Alp12 specification published here [1].
The code builds on the TSM framework patches posted at [2]. We add extension to that framework so that TSM is now used in both the host and the guest.
A DA workflow can be summarized as below:
Quoted 8 lines
Host:
step 1.
echo ${DEVICE} >/sys/bus/pci/devices/${DEVICE}/driver/unbindecho vfio-pci >/sys/bus/pci/devices/${DEVICE}/driver_overrideecho ${DEVICE} >/sys/bus/pci/drivers_probestep 2.
echo 1 >/sys/bus/pci/devices/$DEVICE/tsm/connect
Just for my own understanding... presumably there is no ordering constraint for ARM CCA between step1 and step2, right? I.e. The connect state is independent of the bind state.
In the v4 PCI/TSM scheme the connect command is now:
echo $tsm_dev > /sys/bus/pci/devices/$DEVICE/tsm/connect
Quoted 1 line
Now in the guest we follow the below steps
I assume a signifcant amount of kvmtool magic happens here to get the TDI into a "bind capable" state, can you share that command?
I had been assuming that everyone was prototyping with QEMU. Not a problem per se, but the memory management for shared device assignment / bounce buffering has had a quite of bit of work on the QEMU side, so just curious about the difference in approach here. Like, does kvmtool support operating the device in shared mode with bounce buffering and page conversion (shared <=> private) support? In any event, happy to see mutiple simultaneous consumers of this new kernel infrastructure.
Quoted 5 lines
step 1:
echo ${DEVICE} >/sys/bus/pci/devices/${DEVICE}/driver/unbindstep 2: Move the device to TDISP LOCK state
echo 1 >/sys/bus/pci/devices/${DEVICE}/tsm/lock
Ok, so my stance has recently picked up some nuance here. As Jason mentions here:
http://lore.kernel.org/[email protected]
"However it works, it should be done before the driver is probed and remain stable for the duration of the driver attachment. From the iommu side the correct iommu domain, on the correct IOMMU instance to handle the expected traffic should be setup as the DMA API's iommu domain."
I agree with that up until the point where the implication is userspace control of the UNLOCKED->LOCKED transition. That transition requires enabling bus-mastering (BME), configuring the device into an expected state, and *then* locking the device. That means userspace is blindly hoping that the device is in a state where it will remain quiet on the bus between BME and LOCKED, and that the previous unbind left the device in a state where it is prepared to be locked again.
The BME concern may be overblown given major PCI drivers blindly set BME without validating the device is in a quiesced state, but the "device is prepped for locking" problem seems harder.
2 potential ways to solve this, but open to other ideas:
- Userspace only picks the iommu domain context for the device not the lock state. Something like:
private > /sys/bus/pci/devices/${DEVICE}/tsm/domain ...where the default is "shared" and from that point the device can not issue DMA until a driver attaches. Driver controls UNLOCKED->LOCKED->RUN.
- Userspace is not involved in this transition and the dma mapping API is updated to allow a driver to switch the iommu domain at runtime, but only if the device has no outstanding mappings and the transition can only happen from ->probe() context. Driver controls joining secure-world-DMA and UNLOCKED->LOCKED->RUN.
Clearly the first option is less work in the kernel, but in both options the driver is in control of when BME is set relative to being ready for the LOCKED transition.
Quoted 2 lines
step 3: Moves the device to TDISP RUN state echo 1 >
/sys/bus/pci/devices/${DEVICE}/tsm/accept
This has the same concern from me about userspace being in control of BME. It feels like a departure from typical expectations. At least in the case of a driver setting BME the driver's probe routine is going to get the device in order shortly and otherwise have error handlers at the ready to effect any needed recovery.
Userspace just leaves the device enabled indefinitely and hopes.
Now, the nice thing about the scheme as proposed in this set is that userspace has all the time in the world between "lock" and "accept" to talk to a verifier.
With the driver in control there would need to be something like a usermodehelper to notify userspace that the device is in the locked state and to go ahead and run the attestation while the driver waits*.
* or driver could decide to not wait, especially useful for debug and development
Quoted 2 lines
step 4: Load the driver again.
echo ${DEVICE} >/sys/bus/pci/drivers_probe
TIL drivers_probe
Maybe want to recommend:
echo ${DEVICE} > /sys/bus/pci/drivers/${DRIVER}/bind
...to users just in case there are multiple drivers loaded for the device for the "shared" vs "private" case?
Quoted 2 lines
I'm currently working against TSM v3, as TSM v4 lacks the necessary callbacks—bind, unbind, and guest_req—required for guest interactions.
For staging purposes I wanted to put the "connect" flow to bed before moving on to the guest side.
Quoted 16 lines
The implementation also makes use of RHI interfaces that fall outside the current RHI specification [5]. Once the spec is finalized, the code will be aligned accordingly.
For now, I’ve retained
validate_mmioandvdev_reqexit handling within KVM. This will transition to a guest_req-based mechanism once the specification is updated.At that point, all device assignment (DA)-specific VM exits will exit directly to the VMM, and will use the
guest_reqioctl to handle exit reasons. As part of this change, the handlersrealm_exit_vdev_req_handler,realm_exit_vdev_comm_handler, andrealm_exit_dev_mem_map_handlerwill be removed.Full patchset for the kernel and kvmtool can be found at [3] and [4]
Quoted 7 lines
[1] https://developer.arm.com/-/cdn-downloads/permalink/Architectures/Armv9/DEN0137_1.1-alp12.zip
[2] https://lore.kernel.org/all/[email protected]
[3] https://git.gitlab.arm.com/linux-arm/linux-cca.git cca/tdisp-upstream-post-v1
[4] https://git.gitlab.arm.com/linux-arm/kvmtool-cca.git cca/tdisp-upstream-post-v1
[5] https://developer.arm.com/documentation/den0148/latest/
Thanks for this and the help reviewing PCI/TSM so far! I want to get this into tsm.git#staging so we can start to make hard claims ("look at the shared tree!") of hardware vendor consensus.
Quoted 10 lines · On Thu, Jul 31, 2025 at 07:07:17PM -0700, [email protected] wrote:
Quoted 8 lines · Aneesh Kumar K.V (Arm)
Host:
step 1.
echo ${DEVICE} >/sys/bus/pci/devices/${DEVICE}/driver/unbindecho vfio-pci >/sys/bus/pci/devices/${DEVICE}/driver_overrideecho ${DEVICE} >/sys/bus/pci/drivers_probestep 2.
echo 1 >/sys/bus/pci/devices/$DEVICE/tsm/connect
Quoted 7 lines
Just for my own understanding... presumably there is no ordering constraint for ARM CCA between step1 and step2, right? I.e. The connect state is independent of the bind state.
In the v4 PCI/TSM scheme the connect command is now:
echo $tsm_dev >
/sys/bus/pci/devices/$DEVICE/tsm/connect
What does this do on the host? It seems to somehow prep it for VM assignment? Seems pretty strange this is here in sysfs and not part of creating the vPCI function in the VM through VFIO and iommufd?
Frankly, I'm nervous about making any uAPI whatsoever for the hypervisor side at this point. I don't think we have enough of the solution even in draft format. I'd really like your first merged TSM series to only have uAPI for the guest side where things are hopefully closer to complete..
Quoted 9 lines
Quoted 5 lines
step 1:
echo ${DEVICE} >/sys/bus/pci/devices/${DEVICE}/driver/unbindstep 2: Move the device to TDISP LOCK state
echo 1 >/sys/bus/pci/devices/${DEVICE}/tsm/lockOk, so my stance has recently picked up some nuance here. As Jason mentions here:
Quoted 7 lines
http://lore.kernel.org/[email protected]
"However it works, it should be done before the driver is probed and remain stable for the duration of the driver attachment. From the iommu side the correct iommu domain, on the correct IOMMU instance to handle the expected traffic should be setup as the DMA API's iommu domain."
I think it is not just the dma api, but also the MMIO registers may move location (form shared to protected IPA space for example). Meaning any attached driver is completely wrecked.
Quoted 3 lines
I agree with that up until the point where the implication is userspace control of the
UNLOCKED->LOCKEDtransition. That transition requires enabling bus-mastering (BME),
Why? That's sad. BME should be controlled by the VM driver not the TSM, and it should be set only when a VM driver is probed to the RUN state device?
Quoted 4 lines
and *then* locking the device. That means userspace is blindly hoping that the device is in a state where it will remain quiet on the bus between BME and LOCKED, and that the previous unbind left the device in a state where it is prepared to be locked again.
Yes, but we broadly assume this already in Linux. Drivers assume their devices are quiet when they are bound the first time, we expect on unbinding a driver quiets the device before removing.
So broadly I think you can assume that a device with no driver is quiet regardless of BME.
Quoted 10 lines
2 potential ways to solve this, but open to other ideas:
- Userspace only picks the iommu domain context for the device not the lock state. Something like:
private > /sys/bus/pci/devices/${DEVICE}/tsm/domain...where the default is "shared" and from that point the device can not issue DMA until a driver attaches. Driver controls
UNLOCKED->LOCKED->RUN.
What? Gross, no way can we let userspace control such intimate details of the kernel. The kernel must auto set based on what T=x mode the device driver binds into.
Quoted 5 lines
- Userspace is not involved in this transition and the dma mapping API is updated to allow a driver to switch the iommu domain at runtime, but only if the device has no outstanding mappings and the transition can only happen from ->probe() context. Driver controls joining secure-world-DMA and
UNLOCKED->LOCKED->RUN.
I don't see why it is so complicated. The driver is unbound before it reaches T=1 so we expect the device to be quiet (bigger problems if not). When the PCI core reaches T=1 it tells the DMA API to reconfigure things for the unbound struct device. Then we bind a driver as normal.
Driver controls nothing. All existing T=0 drivers "just work" with no source changes in T=1 mode. DMA API magically hides the bounce buffering. Surely this should be the baseline target functionality from a Linux perspective?
So we should not have "driver controls" statements at all. Userspace prepares the PCI device, driver probes onto a T=1 environment and just works.
Quoted 5 lines
Quoted 2 lines
step 3: Moves the device to TDISP RUN state echo 1 >
/sys/bus/pci/devices/${DEVICE}/tsm/acceptThis has the same concern from me about userspace being in control of BME. It feels like a departure from typical expectations.
It is, it is architecturally broken for BME to be controlled by the TSM. BME is controlled by the guest OS driver only.
IMHO if this is a real worry (and I don't think it is) then the right answer is for physical BME to be set on during locking, but VIRTUAL BME is left off. Virtual BME is created by the hypervisor/tsm by telling the IOMMU to block DMA.
The Guest OS should not participate in this broken design, the hypervisor can set pBME automatically when the lock request comes in, and the quality of vBME emulation is left up to the implementation, but the implementation must provide at least a NOP vBME once locked.
Quoted 3 lines
Now, the nice thing about the scheme as proposed in this set is that userspace has all the time in the world between "lock" and "accept" to talk to a verifier.
Seems right to me. There should be NO trusted kernel driver bound until the verifier accepts the attestation. Anything else allows un accepted devices to attack the kernel drivers. Few kernel drivers today distrust their HW interfaces as hostile actors and security defend against them. Therefore we should be very reluctant to bind drivers to anything..
Arguably a CC secure kernel should have an allow list of audited secure drivers that can autoprobe and all other drivers must be approved by userspace in some way, either through T=1 and attestation or some customer-aware risk assumption.
From that principal the kernel should NOT auto probe drivers to T=0 devices that can be made T=1. Userspace should handle attaching HW to such devices, and userspace can sequence whatever is required, including the attestation and verifying.
Otherwise, if you say, have a TDISP capable mlx5 device and boot up the cVM in a comporomised host the host can probably completely hack your cVM by exploiting the mlx5 drivers's total trust in the HW interface while running in T=0 mode.
You must attest it and switch to T=1 before binding any driver if you care about mitigating this risk.
Quoted 3 lines
With the driver in control there would need to be something like a usermodehelper to notify userspace that the device is in the locked state and to go ahead and run the attestation while the driver waits*.
It doesn't make sense to require modification to all existing drivers in Linux! The starting point must have the core code do this sequence for every driver. Once that is working we can talk about if other flows are needed.
Quoted 11 lines
Quoted 2 lines
step 4: Load the driver again.
echo ${DEVICE} >/sys/bus/pci/drivers_probeTIL
drivers_probeMaybe want to recommend:
echo ${DEVICE} >
/sys/bus/pci/drivers/${DRIVER}/bind...to users just in case there are multiple drivers loaded for the device for the "shared" vs "private" case?
Generic userspace will have a hard time to know what the driver names are..
The driver_probe option looks good to me as the default.
I'm not sure how generic code can handle "multiple drivers".. Most devices will be able to work just fine with T=0 mode with bounce buffers so we should generally not encourage people to make completely different drivers for T=0/T=1 mode.
I think what is needed is some way for userspace to trigger the "locking configuration" you mentioned, that may need a special driver, but ONLY if the userspace is sequencing the device to T=1 mode. Not sure how to make that generic, but I think so long as userspace is explicitly controlling driver binding we can punt on that solution to the userspace project :)
The real nastyness is RAS - what do you do when the device falls out of RUN, the kernel driver should pretty much explode. But lots of people would like the kernel driver to stay alive and somehow we FLR, re-attest and "resume" the kernel driver without allowing any T=0 risks. For instance you can keep your netdev and just see a lot of lost packets while the driver thrashes.
But I think we can start with the idea that such RAS failures have to reload the driver too and work on improvements. Realistically few drivers have the sort of RAS features to consume this anyhow and maybe we introduce some "enhanced" driver mode to opt-into down the road.
Jason
Quoted 15 lines · Jason Gunthorpe
Quoted 14 lines · On Thu, Jul 31, 2025 at 07:07:17PM -0700, [email protected] wrote:
Quoted 8 lines · Aneesh Kumar K.V (Arm)
Host:
step 1.
echo ${DEVICE} >/sys/bus/pci/devices/${DEVICE}/driver/unbindecho vfio-pci >/sys/bus/pci/devices/${DEVICE}/driver_overrideecho ${DEVICE} >/sys/bus/pci/drivers_probestep 2.
echo 1 >/sys/bus/pci/devices/$DEVICE/tsm/connectJust for my own understanding... presumably there is no ordering constraint for ARM CCA between step1 and step2, right? I.e. The connect state is independent of the bind state.
Quoted 7 lines
Quoted 3 lines
In the v4 PCI/TSM scheme the connect command is now:
echo $tsm_dev >
/sys/bus/pci/devices/$DEVICE/tsm/connectWhat does this do on the host? It seems to somehow prep it for VM assignment? Seems pretty strange this is here in sysfs and not part of creating the vPCI function in the VM through VFIO and iommufd?
vPCI is out of the picture at this phase.
On the host this establishes an SPDM session and sets up link encryption (IDE) with the physical device. Leave VMs out of the picture, this capability in isolation is a useful property. It addresses the similar threat model that Intel Total Memory Encryption (TME) or AMD Secure Memory Encryption (SME) go after, i.e. interposer on a physical link capturing data in flight.
With that established then one can go futher to do the full TDISP dance.
Quoted 5 lines
Frankly, I'm nervous about making any uAPI whatsoever for the hypervisor side at this point. I don't think we have enough of the solution even in draft format. I'd really like your first merged TSM series to only have uAPI for the guest side where things are hopefully closer to complete.
Aligned. I am not comfortable merging any of this until we have that end to end reliably stable for a kernel cycle or 2. The proposal is soak all the vendor solutions together in tsm.git#staging.
Now, if the guest side graduates out of that staging before the host side, I am ok with that.
Quoted 17 lines
Quoted 9 lines
Quoted 5 lines
step 1:
echo ${DEVICE} >/sys/bus/pci/devices/${DEVICE}/driver/unbindstep 2: Move the device to TDISP LOCK state
echo 1 >/sys/bus/pci/devices/${DEVICE}/tsm/lockOk, so my stance has recently picked up some nuance here. As Jason mentions here:
Quoted 7 lines
http://lore.kernel.org/[email protected]
"However it works, it should be done before the driver is probed and remain stable for the duration of the driver attachment. From the iommu side the correct iommu domain, on the correct IOMMU instance to handle the expected traffic should be setup as the DMA API's iommu domain."
Quoted 3 lines
I think it is not just the dma api, but also the MMIO registers may move location (form shared to protected IPA space for example). Meaning any attached driver is completely wrecked.
True.
Quoted 7 lines
Quoted 3 lines
I agree with that up until the point where the implication is userspace control of the
UNLOCKED->LOCKEDtransition. That transition requires enabling bus-mastering (BME),Why? That's sad. BME should be controlled by the VM driver not the TSM, and it should be set only when a VM driver is probed to the RUN state device?
To me it is an unfortunate PCI specification wrinkle that writing to the command register drops the device from RUN to ERROR. So you can LOCK without setting BME, but then no DMA.
Quoted 19 lines
Quoted 4 lines
and *then* locking the device. That means userspace is blindly hoping that the device is in a state where it will remain quiet on the bus between BME and LOCKED, and that the previous unbind left the device in a state where it is prepared to be locked again.
Yes, but we broadly assume this already in Linux. Drivers assume their devices are quiet when they are bound the first time, we expect on unbinding a driver quiets the device before removing.
So broadly I think you can assume that a device with no driver is quiet regardless of BME.
Quoted 7 lines
2 potential ways to solve this, but open to other ideas:
- Userspace only picks the iommu domain context for the device not the lock state. Something like:
private > /sys/bus/pci/devices/${DEVICE}/tsm/domain
Quoted 7 lines
Quoted 3 lines
...where the default is "shared" and from that point the device can not issue DMA until a driver attaches. Driver controls
UNLOCKED->LOCKED->RUN.What? Gross, no way can we let userspace control such intimate details of the kernel. The kernel must auto set based on what T=x mode the device driver binds into.
Flummoxed. Any way this gets sliced, userspace is asking for "private world attach" because it alone knows that this device is acceptable, and devices need to arrive in "shared world attach" mode.
Quoted 12 lines
Quoted 5 lines
- Userspace is not involved in this transition and the dma mapping API is updated to allow a driver to switch the iommu domain at runtime, but only if the device has no outstanding mappings and the transition can only happen from ->probe() context. Driver controls joining secure-world-DMA and
UNLOCKED->LOCKED->RUN.I don't see why it is so complicated. The driver is unbound before it reaches T=1 so we expect the device to be quiet (bigger problems if not). When the PCI core reaches T=1 it tells the DMA API to reconfigure things for the unbound struct device. Then we bind a driver as normal.
Quoted 4 lines
Driver controls nothing. All existing T=0 drivers "just work" with no source changes in T=1 mode. DMA API magically hides the bounce buffering. Surely this should be the baseline target functionality from a Linux perspective?
I started this project with "all existing T=0 drivers 'just work'" as a goal and a virtue. I have been begrudgingly pulled away from it from the slow drip of complexity it appears to push into the PCI core.
Now, I suspect the number of devices that are willing to spend gates and firmware on TDISP capabilities in the near term is small. The "just works" case is saved for either an L1 VMM to hide all this from an L2 guest, or a simplified TDISP specification that actually allows an OS PCI core to handle these details in a standard way.
Quoted 3 lines
So we should not have "driver controls" statements at all. Userspace prepares the PCI device, driver probes onto a T=1 environment and just works.
The concern is neither userspace nor the PCI core have everything it needs to get the device to T=1. PCI core knows that the device is T=1 capable, but does not know how to preconfigure the device-specific lock state, needs to wait for attestation. Userpace knows how to attest/verify the device but really has no business running the device outside of binding a driver, and can not rely on the PCI core to have prepped the device's device-specific lock state.
Userspace might be able to bind a new driver that leaves the device in a lockable state on unbind, but that is not "just works" that is, "introduce a new concept of skinny TDISP setup drivers that leave devices in LOCKED state on driver unbind, so that userspace can do the work to verify the device and move it to RUN before loading the main driver that expects the device arrives already running. Also, that main driver needs to be careful not to trigger typically benign actions like touch the command register to trip the device into ERROR state, or any device-specific actions that trip ERROR state but would otherwise be benign outside of TDISP."
If locking the device was just a toggle it would be possible. As far as I can see it is a "prep+toggle" where "prep" needs a driver.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
step 3: Moves the device to TDISP RUN state echo 1 >
/sys/bus/pci/devices/${DEVICE}/tsm/acceptThis has the same concern from me about userspace being in control of BME. It feels like a departure from typical expectations.
It is, it is architecturally broken for BME to be controlled by the TSM. BME is controlled by the guest OS driver only.
Agree. That "accept" attribute does not belong with TSM. That is where Aneesh has it in this RFC. "Accept" as an action is the combination of device entered the LOCKED state in a configuration the verifier is willing to accept and the mechanics of triggering the LOCKED->RUN transition.
Quoted 9 lines
IMHO if this is a real worry (and I don't think it is) then the right answer is for physical BME to be set on during locking, but VIRTUAL BME is left off. Virtual BME is created by the hypervisor/tsm by telling the IOMMU to block DMA.
The Guest OS should not participate in this broken design, the hypervisor can set pBME automatically when the lock request comes in, and the quality of vBME emulation is left up to the implementation, but the implementation must provide at least a NOP vBME once locked.
I can let go of the "BME without driver" worry, but that does nothing to solve the "device specific configuration required before lock" problem.
Quoted 11 lines
Quoted 3 lines
Now, the nice thing about the scheme as proposed in this set is that userspace has all the time in the world between "lock" and "accept" to talk to a verifier.
Seems right to me. There should be NO trusted kernel driver bound until the verifier accepts the attestation. Anything else allows un accepted devices to attack the kernel drivers. Few kernel drivers today distrust their HW interfaces as hostile actors and security defend against them. Therefore we should be very reluctant to bind drivers to anything.
Quoted 4 lines
Arguably a CC secure kernel should have an allow list of audited secure drivers that can autoprobe and all other drivers must be approved by userspace in some way, either through T=1 and attestation or some customer-aware risk assumption.
Yes, today, where nothing is T=1 capable for an L1 guest*, the onus is 100% on the distribution, not the kernel. I.e. trim kernel config and set modprobe policy to prevent unwanted drivers.
* For L2 there are proposals like this, where if you already trust your paravisor also pre-trust all the devices it tells you to trust.
[1]: http://lore.kernel.org/[email protected]
Quoted 4 lines
From that principal the kernel should NOT auto probe drivers to T=0 devices that can be made T=1. Userspace should handle attaching HW to such devices, and userspace can sequence whatever is required, including the attestation and verifying.
Agree, for PCI it would be simple to set a no-auto-probe policy for T=1 capable devices.
Quoted 7 lines
Otherwise, if you say, have a TDISP capable mlx5 device and boot up the cVM in a comporomised host the host can probably completely hack your cVM by exploiting the mlx5 drivers's total trust in the HW interface while running in T=0 mode.
You must attest it and switch to T=1 before binding any driver if you care about mitigating this risk.
Yes, userspace must have a chance to say "no" before a driver attempts to launch DMA to private memory after secrets have been deployed to the TVM.
Quoted 6 lines
Quoted 3 lines
With the driver in control there would need to be something like a usermodehelper to notify userspace that the device is in the locked state and to go ahead and run the attestation while the driver waits*.
It doesn't make sense to require modification to all existing drivers in Linux!
I do not want to burden the PCI core with TDISP compatibility hacks and workarounds if it turns out only a small handful of devices ever deploy a first generation TDISP Device Security Manager (DSM). L1 aiding L2, or TDISP simplicity improvements to allow the PCI core to handle this in a non-broken way, are what I expect if secure device assignment takes off.
Quoted 3 lines
The starting point must have the core code do this sequence for every driver. Once that is working we can talk about if other flows are needed.
Do you agree that "device-specific-prep+lock" is the problem to solve?
Quoted 29 lines
Quoted 11 lines
Quoted 2 lines
step 4: Load the driver again.
echo ${DEVICE} >/sys/bus/pci/drivers_probeTIL
drivers_probeMaybe want to recommend:
echo ${DEVICE} >
/sys/bus/pci/drivers/${DRIVER}/bind...to users just in case there are multiple drivers loaded for the device for the "shared" vs "private" case?
Generic userspace will have a hard time to know what the driver names are..
The
driver_probeoption looks good to me as the default.I'm not sure how generic code can handle "multiple drivers".. Most devices will be able to work just fine with T=0 mode with bounce buffers so we should generally not encourage people to make completely different drivers for T=0/T=1 mode.
I think what is needed is some way for userspace to trigger the "locking configuration" you mentioned, that may need a special driver, but ONLY if the userspace is sequencing the device to T=1 mode. Not sure how to make that generic, but I think so long as userspace is explicitly controlling driver binding we can punt on that solution to the userspace project :)
Quoted 6 lines
The real nastyness is RAS - what do you do when the device falls out of RUN, the kernel driver should pretty much explode. But lots of people would like the kernel driver to stay alive and somehow we FLR, re-attest and "resume" the kernel driver without allowing any T=0 risks. For instance you can keep your netdev and just see a lot of lost packets while the driver thrashes.
Ideally the RUN->ERROR->UNLOCKED->LOCKED->RUN recovery can fit into the existing 'struct pci_error_handlers' regime in some farther out future.
It was a "fun" discovery to see that virtual AER injection does not exist in QEMU (at least last time I checked) and assigned devices that throw physical AER events just kill the VM.
Quoted 4 lines
But I think we can start with the idea that such RAS failures have to reload the driver too and work on improvements. Realistically few drivers have the sort of RAS features to consume this anyhow and maybe we introduce some "enhanced" driver mode to opt-into down the road.
Hmm, having trouble not reading that back supporting my argument above:
Realistically few devices support TDISP lets require enhanced drivers to opt-into TDISP for the time being.
Quoted 6 lines · On Fri, Aug 01, 2025 at 02:19:54PM -0700, [email protected] wrote:
On the host this establishes an SPDM session and sets up link encryption (IDE) with the physical device. Leave VMs out of the picture, this capability in isolation is a useful property. It addresses the similar threat model that Intel Total Memory Encryption (TME) or AMD Secure Memory Encryption (SME) go after, i.e. interposer on a physical link capturing data in flight.
Okay, maybe connect is not an intuitive name for opening IDE sessions..
Quoted 3 lines
I started this project with "all existing T=0 drivers 'just work'" as a goal and a virtue. I have been begrudgingly pulled away from it from the slow drip of complexity it appears to push into the PCI core.
Do you have some examples? I don't really see what complexity there is if the solution it simply not auto bind any drivers to TDISP capable devices and userspace is responsible to manually bind a driver once it has reached T=1.
This seems like the minimum possible simplicitly for the kernel as simply everything is managed by userspace, and there is really no special kernel behavior beyond switching the DMA API of an unbound driver on the T=0/1 change.
Quoted 2 lines
The concern is neither userspace nor the PCI core have everything it needs to get the device to T=1.
Disagree, I think userspace can have everything. It may need some per-device userspace support in difficult cases, but userspace can deal with it..
Quoted 2 lines
PCI core knows that the device is T=1 capable, but does not know how to preconfigure the device-specific lock state,
Userspace can do this. Can we define exactly what is needed to do this "pre-configure the device specific lock state"? At the very worst, for the most poorly designed device, userspace would have to bind a T=0 driver and then unbind it.
Again, I am trying to make something simple for the kernel that gets us to a working solution before we jump ahead to far more complex in the kernel models, like aware drivers that can toggle themselves between T=0/1.
Quoted 2 lines
Userspace might be able to bind a new driver that leaves the device in a lockable state on unbind, but that is not "just works" that is,
I wouldn't have the kernel leave the device in the locked state. That should always be userspace. The special driver may do whatever special setup is needed, then unbind and leave a normal unlocked device "prepped" for userspace locking without doing a FLR or something. Realistically I expect this to be a very rare requirement, I think this coming up just reflects the HW immaturity of some early TDISP devices.
Sensible mature devices should have no need of a pre-locking step. I think we should design toward that goal as the stable future and only try to enable a hacky work around for the problematic early devices. I certainly am not keen on seeing significant permanent kernel complexity to support this device design defect.
Quoted 5 lines
driver that expects the device arrives already running. Also, that main driver needs to be careful not to trigger typically benign actions like touch the command register to trip the device into ERROR state, or any device-specific actions that trip ERROR state but would otherwise be benign outside of TDISP."
As I said below, I disagree with this. You can't touch the *physical* command register but the cVM can certainly touch the *virtualized* command register. It up to the VMM To ensure this doesn't cause the device to fall out of RUN as part of virtualization.
I'd also say that the VMM should be responsible to set pBME=1 even if vBME=0? Shouldn't it? That simplifies even more things for the guest.
Quoted 7 lines
Quoted 4 lines
From that principal the kernel should NOT auto probe drivers to T=0 devices that can be made T=1. Userspace should handle attaching HW to such devices, and userspace can sequence whatever is required, including the attestation and verifying.
Agree, for PCI it would be simple to set a no-auto-probe policy for T=1 capable devices.
So then it is just a question of what does a userspace component need to do.
Quoted 5 lines
I do not want to burden the PCI core with TDISP compatibility hacks and workarounds if it turns out only a small handful of devices ever deploy a first generation TDISP Device Security Manager (DSM). L1 aiding L2, or TDISP simplicity improvements to allow the PCI core to handle this in a non-broken way, are what I expect if secure device assignment takes off.
Same feeling about pre-configuration :)
Quoted 5 lines
Quoted 3 lines
The starting point must have the core code do this sequence for every driver. Once that is working we can talk about if other flows are needed.
Do you agree that "device-specific-prep+lock" is the problem to solve?
Not "the" problem, but an design issue we need to accommodate but not endorse.
Quoted 9 lines
Quoted 4 lines
But I think we can start with the idea that such RAS failures have to reload the driver too and work on improvements. Realistically few drivers have the sort of RAS features to consume this anyhow and maybe we introduce some "enhanced" driver mode to opt-into down the road.
Hmm, having trouble not reading that back supporting my argument above:
Realistically few devices support TDISP lets require enhanced drivers to opt-into TDISP for the time being.
I would be comfortable if hitless RAS recovery for TDISP devices requires some kernel opt-in. But also I'm not sure how this should work from a security perspective. Should userspace also have to re-attest before allowing back to RUN? Clearly this is complicated.
Also, I would be comfortable to support this only for devices that do not require pre-configuration.
Jason
Quoted 11 lines · Jason Gunthorpe
Quoted 6 lines · On Fri, Aug 01, 2025 at 02:19:54PM -0700, [email protected] wrote:
On the host this establishes an SPDM session and sets up link encryption (IDE) with the physical device. Leave VMs out of the picture, this capability in isolation is a useful property. It addresses the similar threat model that Intel Total Memory Encryption (TME) or AMD Secure Memory Encryption (SME) go after, i.e. interposer on a physical link capturing data in flight.
Okay, maybe connect is not an intuitive name for opening IDE sessions..
Part of the rationale for a generic name is the TSM is free to assert that the link is secure without IDE. Think integrated devices where there is no expectation the link can be observed.
The host and guest side TSM operations are split into link/transport security and device/state security (private MMIO/DMA) concerns respectively. So maybe "secure_link" would be a better name for this host-side-only operation.
Quoted 8 lines
Quoted 3 lines
I started this project with "all existing T=0 drivers 'just work'" as a goal and a virtue. I have been begrudgingly pulled away from it from the slow drip of complexity it appears to push into the PCI core.
Do you have some examples? I don't really see what complexity there is if the solution it simply not auto bind any drivers to TDISP capable devices and userspace is responsible to manually bind a driver once it has reached T=1.
The example I have front of mind (confirmed by 2 vendors) is deferring the loading of guest-side device/state security capable firmware to the guest driver when the full device is assigned. In that scenario default device power-on firmware is capable of link/transport security, enough to get the device assigned. Guest needs to get the device/state security firmware loaded before TDISP state transitions are possible.
I do think RAS recovery needs it too, but like you say below that should come with conditions.
Quoted 11 lines
This seems like the minimum possible simplicitly for the kernel as simply everything is managed by userspace, and there is really no special kernel behavior beyond switching the DMA API of an unbound driver on the T=0/1 change.
Quoted 2 lines
The concern is neither userspace nor the PCI core have everything it needs to get the device to T=1.
Disagree, I think userspace can have everything. It may need some per-device userspace support in difficult cases, but userspace can deal with it..
I do think userspace can / must deal with it. Let me come back with actual patches and a sample test case. I see a potential path to support the above "prep" scenario without the mess of TDISP setup drivers, or the ugly complexity of driver toggles or a usermodehelper.
Quoted 8 lines
Quoted 2 lines
PCI core knows that the device is T=1 capable, but does not know how to preconfigure the device-specific lock state,
Userspace can do this. Can we define exactly what is needed to do this "pre-configure the device specific lock state"? At the very worst, for the most poorly designed device, userspace would have to bind a T=0 driver and then unbind it.
Quoted 4 lines
Again, I am trying to make something simple for the kernel that gets us to a working solution before we jump ahead to far more complex in the kernel models, like aware drivers that can toggle themselves between T=0/1.
Agree. When I talked about wishing for the simple TDISP case that is userspace can always "just lock" and "driver bind" without needing to worry about "prep", i.e any "prep" is always implied by "lock". That should be the baseline.
Quoted 11 lines
Quoted 2 lines
Userspace might be able to bind a new driver that leaves the device in a lockable state on unbind, but that is not "just works" that is,
I wouldn't have the kernel leave the device in the locked state. That should always be userspace. The special driver may do whatever special setup is needed, then unbind and leave a normal unlocked device "prepped" for userspace locking without doing a FLR or something. Realistically I expect this to be a very rare requirement, I think this coming up just reflects the HW immaturity of some early TDISP devices.
Quoted 5 lines
Sensible mature devices should have no need of a pre-locking step. I think we should design toward that goal as the stable future and only try to enable a hacky work around for the problematic early devices. I certainly am not keen on seeing significant permanent kernel complexity to support this device design defect.
Yeah, that is the nightmare I had last night. I completed the thought exercise about driver toggle and said, "whoops, nope, Jason is right, we can't design for that without leaving a permanent mess to cleanup". The end goal needs to look like straight line typical driver probe path for TDISP capable devices.
Quoted 6 lines
Quoted 5 lines
driver that expects the device arrives already running. Also, that main driver needs to be careful not to trigger typically benign actions like touch the command register to trip the device into ERROR state, or any device-specific actions that trip ERROR state but would otherwise be benign outside of TDISP."
Quoted 7 lines
As I said below, I disagree with this. You can't touch the *physical* command register but the cVM can certainly touch the *virtualized* command register. It up to the VMM To ensure this doesn't cause the device to fall out of RUN as part of virtualization.
I'd also say that the VMM should be responsible to set pBME=1 even if vBME=0? Shouldn't it? That simplifies even more things for the guest.
True. Although, now I am going back on my PCI core burden concern to wonder if *it* should handle a vBME on behalf of the driver if only because it may want to force the device out of the RUN state on driver unbind to meet typical pci_disable_device() expectations.
Alexey had this, I thought it was burdensome, now coming around.
Quoted 19 lines
Quoted 7 lines
Quoted 4 lines
From that principal the kernel should NOT auto probe drivers to T=0 devices that can be made T=1. Userspace should handle attaching HW to such devices, and userspace can sequence whatever is required, including the attestation and verifying.
Agree, for PCI it would be simple to set a no-auto-probe policy for T=1 capable devices.
So then it is just a question of what does a userspace component need to do.
Quoted 5 lines
I do not want to burden the PCI core with TDISP compatibility hacks and workarounds if it turns out only a small handful of devices ever deploy a first generation TDISP Device Security Manager (DSM). L1 aiding L2, or TDISP simplicity improvements to allow the PCI core to handle this in a non-broken way, are what I expect if secure device assignment takes off.
Same feeling about pre-configuration :)
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
The starting point must have the core code do this sequence for every driver. Once that is working we can talk about if other flows are needed.
Do you agree that "device-specific-prep+lock" is the problem to solve?
Not "the" problem, but an design issue we need to accommodate but not endorse.
I hear you, let me walk back from the cliff with patches.
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
But I think we can start with the idea that such RAS failures have to reload the driver too and work on improvements. Realistically few drivers have the sort of RAS features to consume this anyhow and maybe we introduce some "enhanced" driver mode to opt-into down the road.
Hmm, having trouble not reading that back supporting my argument above:
Realistically few devices support TDISP lets require enhanced drivers to opt-into TDISP for the time being.
Quoted 7 lines
I would be comfortable if hitless RAS recovery for TDISP devices requires some kernel opt-in. But also I'm not sure how this should work from a security perspective. Should userspace also have to re-attest before allowing back to RUN? Clearly this is complicated.
Also, I would be comfortable to support this only for devices that do not require pre-configuration.
That seems reasonable. You want hitless RAS? Give us hitless init.
Quoted 11 lines · On Sat, Aug 02, 2025 at 04:50:50PM -0700, [email protected] wrote:
Quoted 4 lines
Do you have some examples? I don't really see what complexity there is if the solution it simply not auto bind any drivers to TDISP capable devices and userspace is responsible to manually bind a driver once it has reached T=1.
The example I have front of mind (confirmed by 2 vendors) is deferring the loading of guest-side device/state security capable firmware to the guest driver when the full device is assigned. In that scenario default device power-on firmware is capable of link/transport security, enough to get the device assigned. Guest needs to get the device/state security firmware loaded before TDISP state transitions are possible.
Yeah, those are the only cases I know of too, and IMHO, they are just early devices. Clearly the clean answer is to put enough boot FW on the device's flash to get to T=1 mode, then have the trusted OS driver load the operating firmware from the trusted OS filesystem though the trusted bootloader T=1 device.
You effectively attest the bootloader, and then if you trust the bootloader you know that when the device gets to T=1 it can be trusted to properly run the FW the trusted driver provides.
Think about this more broadly, does the prep FW load idea make sense for something like SRIOV? No, it really doesn't. The hypervisor loaded FW that is running the PF should definately be strong enough to get to T=1 on the VM/VF side as well.
The non-SRIOV cases are quite often whole machine assignment scenarios. But I'm sensing alot of that space is moving toward bare metal machines instead of VMs.
I wonder if you can use all the CC machinery to attest and secure a bare metal host?
Quoted 2 lines
I do think RAS recovery needs it too, but like you say below that should come with conditions.
Especially RAS becomes simple because it basically follows the normal flows that existed prior to TDISP, with the exception of needing some attestation step.
I don't know alot about CC attestation, but maybe we can have userspace provide the kernel with the accepted measurement and then for RAS the kernel can FLR, remeasure and if the measurement is exactly the same go back into T=1 automatically as part of the PCI core FLR logic.
Quoted 4 lines
I do think userspace can / must deal with it. Let me come back with actual patches and a sample test case. I see a potential path to support the above "prep" scenario without the mess of TDISP setup drivers, or the ugly complexity of driver toggles or a usermodehelper.
I don't see how, something nasty has to be done in the kernel to allow an attached driver to switch between T=1 and T=0 "views" of the device and lockstep those changes with userspace. This is not so simple and it really basically exactly the same as driver binding.
I don't think we should be afraid of T=0 prep drivers in these early days.
Something more complex could come later if it is really warranted and people really insist on continuing this unclean device design strategy.
Quoted 5 lines
Yeah, that is the nightmare I had last night. I completed the thought exercise about driver toggle and said, "whoops, nope, Jason is right, we can't design for that without leaving a permanent mess to cleanup". The end goal needs to look like straight line typical driver probe path for TDISP capable devices.
Yeah, maybe it is worthwhile to someday try to figure out an alternative - keep in mind that critically this requires someone to also come with an intree driver that will use all these new APIs and capabilities!!!
So lets get walking first and then someone can come with some proposal, complete with a driver implementing it, and it can be judged. This project is already so big, and I'm pretty sure if you start to also need entirely new operating modes for drivers the basics will just get bogged down in that discussion, and very likely killed anyhow due to a lack of user.
Even if we decide that is prefered it is better to separate it and discuss it after the basics are merged. At least where I sit getting basic guest support is a big priority so I strongly want to strip it down to minimal as possible to make consistent progress steps.
Quoted 4 lines
True. Although, now I am going back on my PCI core burden concern to wonder if *it* should handle a vBME on behalf of the driver if only because it may want to force the device out of the RUN state on driver unbind to meet typical
pci_disable_device()expectations.
Hiding some vBME in the PCI core might make sense if we can't get the VMM owners to agree to do it on the hypervisor side. It works better on the VMM side because there is always an IOMMU and the VMM can emulate BME by blocking DMA with the IOMMU.
But I would not allow/expect kernel device drivers to have anything to do with the TDISP states. Getting into RUN is fully sequenced by userspace, getting out of run should also be sequenced only by userspace.
Removing a driver does not change the trust state of the PCI device, so it shouldn't drop out of RUN. If userspace wishes to FLR the device after userspace asked to unbind it can, there are already sysfs controls for this IIRC.
Basically, all this says that Linux drivers that want to be used with T=1 should be well behaved, fully quite all their DMA on remove, and have no *functional* need for BME to do anyhting. We pretty much already expect this of drivers today, so I don't see an issue with strongly requiring it for T=1.
Keep in mind the flip side, almost no drivers are structured properly to forcibly quiet any DMA before pci_enable_device(). Some HW, like mlx5, can't do this at all without either using DMA to send a reset command or through FLR.
Quoted 9 lines
Quoted 7 lines
I would be comfortable if hitless RAS recovery for TDISP devices requires some kernel opt-in. But also I'm not sure how this should work from a security perspective. Should userspace also have to re-attest before allowing back to RUN? Clearly this is complicated.
Also, I would be comfortable to support this only for devices that do not require pre-configuration.
That seems reasonable. You want hitless RAS? Give us hitless init.
Yeah.. Realistically there are few drivers that can even do this today, mlx5 for example has such code (and it is hard!).
There is alot of investment required in the driver's core subsystem to make this work. netdev and RDMA can support a 'rebirth' sort of flow where the driver can disconnect the SW APIs, FLR the device, then reconnect in some way. However, for example, I recently had a discussion with DRM guys about RAS and they are not even doing the basic locking/etc to be able to do this. :\
Jason
Aneesh Kumar K.V
maintainerQuoted 75 lines · <[email protected]> writes:
Quoted 15 lines · Jason Gunthorpe
Quoted 14 lines · On Thu, Jul 31, 2025 at 07:07:17PM -0700, [email protected] wrote:
Quoted 8 lines · Aneesh Kumar K.V (Arm)
Host:
step 1.
echo ${DEVICE} >/sys/bus/pci/devices/${DEVICE}/driver/unbindecho vfio-pci >/sys/bus/pci/devices/${DEVICE}/driver_overrideecho ${DEVICE} >/sys/bus/pci/drivers_probestep 2.
echo 1 >/sys/bus/pci/devices/$DEVICE/tsm/connectJust for my own understanding... presumably there is no ordering constraint for ARM CCA between step1 and step2, right? I.e. The connect state is independent of the bind state.
Quoted 7 lines
Quoted 3 lines
In the v4 PCI/TSM scheme the connect command is now:
echo $tsm_dev >
/sys/bus/pci/devices/$DEVICE/tsm/connectWhat does this do on the host? It seems to somehow prep it for VM assignment? Seems pretty strange this is here in sysfs and not part of creating the vPCI function in the VM through VFIO and iommufd?
vPCI is out of the picture at this phase.
On the host this establishes an SPDM session and sets up link encryption (IDE) with the physical device. Leave VMs out of the picture, this capability in isolation is a useful property. It addresses the similar threat model that Intel Total Memory Encryption (TME) or AMD Secure Memory Encryption (SME) go after, i.e. interposer on a physical link capturing data in flight.
With that established then one can go futher to do the full TDISP dance.
Quoted 5 lines
Frankly, I'm nervous about making any uAPI whatsoever for the hypervisor side at this point. I don't think we have enough of the solution even in draft format. I'd really like your first merged TSM series to only have uAPI for the guest side where things are hopefully closer to complete.
Aligned. I am not comfortable merging any of this until we have that end to end reliably stable for a kernel cycle or 2. The proposal is soak all the vendor solutions together in tsm.git#staging.
Now, if the guest side graduates out of that staging before the host side, I am ok with that.
Quoted 17 lines
Quoted 9 lines
Quoted 5 lines
step 1:
echo ${DEVICE} >/sys/bus/pci/devices/${DEVICE}/driver/unbindstep 2: Move the device to TDISP LOCK state
echo 1 >/sys/bus/pci/devices/${DEVICE}/tsm/lockOk, so my stance has recently picked up some nuance here. As Jason mentions here:
Quoted 7 lines
http://lore.kernel.org/[email protected]
"However it works, it should be done before the driver is probed and remain stable for the duration of the driver attachment. From the iommu side the correct iommu domain, on the correct IOMMU instance to handle the expected traffic should be setup as the DMA API's iommu domain."
Quoted 3 lines
I think it is not just the dma api, but also the MMIO registers may move location (form shared to protected IPA space for example). Meaning any attached driver is completely wrecked.
True.
Quoted 4 lines
Quoted 3 lines
I agree with that up until the point where the implication is userspace control of the
UNLOCKED->LOCKEDtransition. That transition requires enabling bus-mastering (BME),
Quoted 8 lines
Quoted 3 lines
Why? That's sad. BME should be controlled by the VM driver not the TSM, and it should be set only when a VM driver is probed to the RUN state device?
To me it is an unfortunate PCI specification wrinkle that writing to the command register drops the device from RUN to ERROR. So you can LOCK without setting BME, but then no DMA.
This is only w.r.t clearing BME isn't ?
According to section 11.2.6 DSM Tracking and Handling of Locked TDI Configurations
Clearing any of the following bits causes the TDI hosted by the Function to transition to ERROR:
• Memory Space Enable
• Bus Master Enable
Which implies the flow described in the cover-letter where driver enable the BME works? However clearing BME may be problematic? I did have a FIXME!!/comment in [1]
vfio_pci_core_close_device():
#if 0 /* * destroy vdevice which involves tsm unbind before we disable pci disable * A MSE/BME clear will transition the device to error state. */ if (core_vdev->iommufd_device) iommufd_device_tombstone_vdevice(core_vdev->iommufd_device);
#endif
vfio_pci_core_disable(vdev);
Currently, we destroy (TSM unbind) the vdevice after calling vfio_pci_core_disable(), which means BME is cleared before unbinding, and the TDI transitions to the ERROR state.
[1] https://lore.kernel.org/all/[email protected]/
-aneesh
Quoted 6 lines · Aneesh Kumar K.V
Quoted 3 lines
To me it is an unfortunate PCI specification wrinkle that writing to the command register drops the device from RUN to ERROR. So you can LOCK without setting BME, but then no DMA.
This is only w.r.t clearing BME isn't ?
Quoted 7 lines
According to section 11.2.6 DSM Tracking and Handling of Locked TDI Configurations
Clearing any of the following bits causes the TDI hosted by the Function to transition to ERROR:
• Memory Space Enable
• Bus Master Enable
Oh that's nice, yeah!
Quoted 12 lines
Which implies the flow described in the cover-letter where driver enable the BME works? However clearing BME may be problematic? I did have a FIXME!!/comment in [1]
vfio_pci_core_close_device():#if 0 /* * destroy vdevice which involves tsm unbind before we disable pci disable * A MSE/BME clear will transition the device to error state. */ if (core_vdev->iommufd_device) iommufd_device_tombstone_vdevice(core_vdev->iommufd_device);
Quoted 3 lines
#endif
vfio_pci_core_disable(vdev);
Here is where I feel the VMM should be trapping this and NOPing it, or failing that the guest PCI Core should NOP it.
With the ideal version being the TSM and VMM would be able to block the iommu as a functional stand in for BME.
Quoted 3 lines
Currently, we destroy (TSM unbind) the vdevice after calling
vfio_pci_core_disable(), which means BME is cleared before unbinding, and the TDI transitions to the ERROR state.
I don't think this ordering is deliberate, we can destroy the vdevice much earlier??
Jason
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · Aneesh Kumar K.V
Quoted 3 lines
To me it is an unfortunate PCI specification wrinkle that writing to the command register drops the device from RUN to ERROR. So you can LOCK without setting BME, but then no DMA.
This is only w.r.t clearing BME isn't ?
According to section 11.2.6 DSM Tracking and Handling of Locked TDI Configurations
Quoted 7 lines
Quoted 5 lines
Clearing any of the following bits causes the TDI hosted by the Function to transition to ERROR:
• Memory Space Enable
• Bus Master EnableOh that's nice, yeah!
That is useful, but an unmodified PCI driver is going to make separate calls to pci_set_master() and pci_enable_device() so it should still be the case that those need to be trapped out of the concern that writing back zero for a read-modify-write also trips the error state on some device that fails the Robustness Principle.
I guess we could wait to solve that problem until the encountering the first device that trips ERROR when writing zero to an already zeroed bit.
Quoted 14 lines
Quoted 14 lines
Which implies the flow described in the cover-letter where driver enable the BME works? However clearing BME may be problematic? I did have a FIXME!!/comment in [1]
vfio_pci_core_close_device():#if 0 /* * destroy vdevice which involves tsm unbind before we disable pci disable * A MSE/BME clear will transition the device to error state. */ if (core_vdev->iommufd_device) iommufd_device_tombstone_vdevice(core_vdev->iommufd_device);#endif
Quoted 4 lines
Quoted 1 line
vfio_pci_core_disable(vdev);Here is where I feel the VMM should be trapping this and NOPing it, or failing that the guest PCI Core should NOP it.
At this point (vfio shutdown path) the VMM is committed stopping guest operations with the device. So ok not to not NOP in this specific path, right?
Quoted 2 lines
With the ideal version being the TSM and VMM would be able to block the iommu as a functional stand in for BME.
The TSM block for BME is the LOCKED or ERROR state. That would be in conflict with the proposal that the device stays in the RUN state on guest driver unbind.
I feel like either the device stays in RUN state and BME leaks, or the device is returned to LOCKED on driver unbind. Otherwise a functional stand-in for BME that also keeps the device in RUN state feels like a TSM feature request for a "RUN but BLOCKED" state.
Quoted 6 lines · On Tue, Aug 05, 2025 at 11:27:36AM -0700, [email protected] wrote:
Quoted 6 lines
Quoted 5 lines
Clearing any of the following bits causes the TDI hosted by the Function to transition to ERROR:
• Memory Space Enable
• Bus Master Enable
Quoted 7 lines
Quoted 1 line
Oh that's nice, yeah!
That is useful, but an unmodified PCI driver is going to make separate calls to
pci_set_master()andpci_enable_device()so it should still be the case that those need to be trapped out of the concern that writing back zero for a read-modify-write also trips the error state on some device that fails the Robustness Principle.
I hope we don't RMW BME and MSE in some weird way like that :(
Quoted 6 lines
Quoted 2 lines
Here is where I feel the VMM should be trapping this and NOPing it, or failing that the guest PCI Core should NOP it.
At this point (vfio shutdown path) the VMM is committed stopping guest operations with the device. So ok not to not NOP in this specific path, right?
What I said in my other mail was the the T=1 state should have nothing to do with driver binding. So unbinding vfio should leave the device in the RUN state just fine.
Quoted 6 lines
Quoted 2 lines
With the ideal version being the TSM and VMM would be able to block the iommu as a functional stand in for BME.
The TSM block for BME is the LOCKED or ERROR state. That would be in conflict with the proposal that the device stays in the RUN state on guest driver unbind.
This is a different thing. Leaving RUN says the OS (especially userspace) does not trust the device.
Disabling DMA, on explict trusted request from the cVM, is entirely fine to do inside the T=1 state. PCI made it so the only way to do this is with the IOMMU, oh well, so be it.
Quoted 2 lines
I feel like either the device stays in RUN state and BME leaks, or the device is returned to LOCKED on driver unbind.
Stay in RUN is my vote. I can't really defend the other choice from a linux driver model perspective.
Quoted 3 lines
Otherwise a functional stand-in for BME that also keeps the device in RUN state feels like a TSM feature request for a "RUN but BLOCKED" state.
Yes, and probably not necessary, more of a defence against bugs in depth kind of request. For Linux we would like it if the device can be in RUN and have DMA blocked off during all times when no driver is attached.
Jason
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · On Tue, Aug 05, 2025 at 11:27:36AM -0700, [email protected] wrote:
Quoted 7 lines
Quoted 5 lines
Clearing any of the following bits causes the TDI hosted by the Function to transition to ERROR:
• Memory Space Enable
• Bus Master EnableOh that's nice, yeah!
Quoted 7 lines
Quoted 5 lines
That is useful, but an unmodified PCI driver is going to make separate calls to
pci_set_master()andpci_enable_device()so it should still be the case that those need to be trapped out of the concern that writing back zero for a read-modify-write also trips the error state on some device that fails the Robustness Principle.I hope we don't RMW BME and MSE in some weird way like that :(
Yeah, I would like to say, "device, you get to keep the pieces if you transition to ERROR state on re-writing on already zeroed-bit."
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Here is where I feel the VMM should be trapping this and NOPing it, or failing that the guest PCI Core should NOP it.
At this point (vfio shutdown path) the VMM is committed stopping guest operations with the device. So ok not to not NOP in this specific path, right?
What I said in my other mail was the the T=1 state should have nothing to do with driver binding.
Guest driver unbind, agree.
Quoted 1 line
So unbinding vfio should leave the device in the RUN state just fine.
Perhaps my vfio inexperience is showing, but at the point where the VMM is unbinding vfio it is committed to destroying the guest's assigned device context, no? So should that not be the point where continuing to maintain the RUN state ends?
Quoted 20 lines
Quoted 6 lines
Quoted 2 lines
With the ideal version being the TSM and VMM would be able to block the iommu as a functional stand in for BME.
The TSM block for BME is the LOCKED or ERROR state. That would be in conflict with the proposal that the device stays in the RUN state on guest driver unbind.
This is a different thing. Leaving RUN says the OS (especially userspace) does not trust the device.
Disabling DMA, on explict trusted request from the cVM, is entirely fine to do inside the T=1 state. PCI made it so the only way to do this is with the IOMMU, oh well, so be it.
Quoted 2 lines
I feel like either the device stays in RUN state and BME leaks, or the device is returned to LOCKED on driver unbind.
Stay in RUN is my vote. I can't really defend the other choice from a linux driver model perspective.
Quoted 8 lines
Quoted 3 lines
Otherwise a functional stand-in for BME that also keeps the device in RUN state feels like a TSM feature request for a "RUN but BLOCKED" state.
Yes, and probably not necessary, more of a defence against bugs in depth kind of request. For Linux we would like it if the device can be in RUN and have DMA blocked off during all times when no driver is attached.
Ok, defense in depth, but in the meantime rely on unbound driver == DMA unmapped and device should be quiescent. Combine that with the fact that userspace PCI drivers should be disabled in cVMs should mean that guest can expect that an unbound TDI in the RUN state will remain quiet.
Quoted 6 lines · On Tue, Aug 05, 2025 at 12:06:11PM -0700, [email protected] wrote:
Quoted 1 line
So unbinding vfio should leave the device in the RUN state just fine.
Perhaps my vfio inexperience is showing, but at the point where the VMM is unbinding vfio it is committed to destroying the guest's assigned device context, no? So should that not be the point where continuing to maintain the RUN state ends?
Oh, sorry it gets so confusing..
VFIO *in the guest* should behave as above, like any other driver unbind leaves it in RUN.
VFIO *in the host* should leave the RUN state at the soonest of:
- cVM's KVM is destroyed
- iommufd vdevice is destroyed
- vfio device is closed
And maybe more cases I didn't think of.. BME should happen strictly after all of the above and should not be the trigger that drops it out of RUN.
Quoted 9 lines
Quoted 4 lines
Yes, and probably not necessary, more of a defence against bugs in depth kind of request. For Linux we would like it if the device can be in RUN and have DMA blocked off during all times when no driver is attached.
Ok, defense in depth, but in the meantime rely on unbound driver == DMA unmapped and device should be quiescent. Combine that with the fact that userspace PCI drivers should be disabled in cVMs should mean that guest can expect that an unbound TDI in the RUN state will remain quiet.
"userspace PCI drivers" is VFIO in the guest which means you get FLRs to fence the DMA.
If we end up where I suggested earlier for RAS that a FLR can check the attestation and if exactly matching reaccept it automatically then it would maintain the 'once accepted we stay in T=1 RUN state' idea.
Jason
Aneesh Kumar K.V
maintainerQuoted 23 lines · <[email protected]> writes:
Quoted 9 lines · Aneesh Kumar K.V (Arm)
This patch series implements support for Device Assignment in the ARM CCA architecture. The code changes are based on Alp12 specification published here [1].
The code builds on the TSM framework patches posted at [2]. We add extension to that framework so that TSM is now used in both the host and the guest.
A DA workflow can be summarized as below:
Quoted 8 lines
Host:
step 1.
echo ${DEVICE} >/sys/bus/pci/devices/${DEVICE}/driver/unbindecho vfio-pci >/sys/bus/pci/devices/${DEVICE}/driver_overrideecho ${DEVICE} >/sys/bus/pci/drivers_probestep 2.
echo 1 >/sys/bus/pci/devices/$DEVICE/tsm/connectJust for my own understanding... presumably there is no ordering constraint for ARM CCA between step1 and step2, right? I.e. The connect state is independent of the bind state.
Quoted 9 lines
In the v4 PCI/TSM scheme the connect command is now:
echo $tsm_dev >
/sys/bus/pci/devices/$DEVICE/tsm/connectQuoted 1 line
Now in the guest we follow the below steps
I assume a signifcant amount of kvmtool magic happens here to get the TDI into a "bind capable" state, can you share that command?
lkvm run --realm -c 2 -m 256 -k /kselftest/Image -p "$KERNEL_PARAMS" -d ./rootfs-guest.ext2 --iommufd-vdevice --vfio-pci $DEVICE1 --vfio-pci $DEVICE2
Quoted 8 lines
I had been assuming that everyone was prototyping with QEMU. Not a problem per se, but the memory management for shared device assignment / bounce buffering has had a quite of bit of work on the QEMU side, so just curious about the difference in approach here. Like, does kvmtool support operating the device in shared mode with bounce buffering and page conversion (shared <=> private) support? In any event, happy to see mutiple simultaneous consumers of this new kernel infrastructure.
-aneesh