x86/iommu: Drop IOMMU support when cx16 isn't supported
v2 cover x86/iommu: Drop IOMMU support when cx16 isn't supported
All hardware that supports VT-d/AMD-Vi that exists also supports cx16 (aside specifically crafted virtual machines).
Some IOMMU code paths in Xen consider cases where VT-d/AMD-Vi is supported while cx16 isn't, those paths may be bugged and are barely tested, dead code in practice.
Disable IOMMU in case we have IOMMU hardware but no cx16, then cleanup no-cx16 handling logic from VT-d and AMD-Vi drivers.
Teddy
Changed in v2:
* Added cleanup no-cx16 code for x2APIC
* Fixed commit and code formatting
* Added missing Suggested-by note
Teddy Astie (3):
VT-d: Disable IOMMU if cx16 isn't supported
AMD-Vi: Disable IOMMU if cx16 isn't supported
VT-d: Cleanup MAP_SINGLE_DEVICE and related code
xen/arch/x86/apic.c | 6 ++ xen/drivers/passthrough/amd/iommu_map.c | 42 ++++------ xen/drivers/passthrough/amd/pci_amd_iommu.c | 6 ++ xen/drivers/passthrough/vtd/intremap.c | 65 ++++----------- xen/drivers/passthrough/vtd/iommu.c | 92 +++++++-------------- xen/drivers/passthrough/vtd/vtd.h | 5 +- 6 files changed, 71 insertions(+), 145 deletions(-)
Signature · 9 lines
2.44.0 Teddy Astie | Vates XCP-ng Intern XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
PATCH v2 1/3 VT-d: Disable IOMMU if cx16 isn't supported
No hardware has VT-d support while not having cx16 support, disable IOMMU in this case to avoid potentially buggy code.
Now that IOMMU is only enabled if cx16 is supported, drop dead code that handles cases where cx16 isn't supported.
--- xen/arch/x86/apic.c | 6 ++ xen/drivers/passthrough/vtd/intremap.c | 65 +++++---------------- xen/drivers/passthrough/vtd/iommu.c | 80 +++++++++----------------- 3 files changed, 46 insertions(+), 105 deletions(-)
xen/arch/x86/apic.c +6 -0
@@ -836,6 +836,12 @@ void __init x2apic_bsp_setup(void)if ( !cpu_has_x2apic )return ; +if (unlikely (!cpu_has_cx16) ) + { +printk ("x2APIC: CPU doesn't support CMPXCHG16B, disabling\n" ); +return ; + } +if ( !opt_x2apic ) {if ( !x2apic_enabled )
xen/drivers/passthrough/vtd/intremap.c +14 -51
89 lines not on this page · raw message
xen/drivers/passthrough/vtd/iommu.c +26 -54
106 lines not on this page · raw message
Signature · 9 lines
2.44.0 Teddy Astie | Vates XCP-ng Intern XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
PATCH v2 2/3 AMD-Vi: Disable IOMMU if cx16 isn't supported
In reply to Teddy AstieNo hardware has AMD-Vi support while not having cx16 support, disable IOMMU in this case to avoid potentially buggy code.
Now that IOMMU is only enabled if cx16 is supported, drop dead code that handles cases where cx16 isn't supported.
--- xen/drivers/passthrough/amd/iommu_map.c | 42 +++++++-------------- xen/drivers/passthrough/amd/pci_amd_iommu.c | 6 +++ 2 files changed, 20 insertions(+), 28 deletions(-)
xen/drivers/passthrough/amd/iommu_map.c +14 -28
@@ -167,15 +167,14 @@ int amd_iommu_set_root_page_table(struct amd_iommu_dte *dte, {bool valid = flags &SET_ROOT_VALID ; -if ( dte->v && dte->tv && - (cpu_has_cx16 || (flags &SET_ROOT_WITH_UNITY_MAP )) ) +if ( dte->v && dte->tv ) {union {struct amd_iommu_dte dte;uint64_t raw64[4 ];__uint128_t raw128[2 ]; } ldte = { .dte = *dte }; -__uint128_t old = ldte.raw128[0 ]; +__uint128_t res, old = ldte.raw128[0 ];int ret =0 ; ldte.dte.domain_id = domain_id;
@@ -185,33 +184,20 @@ int amd_iommu_set_root_page_table(struct amd_iommu_dte *dte, ldte.dte.paging_mode = paging_mode; ldte.dte.v = valid; -if ( cpu_has_cx16 ) - { -__uint128_t res =cmpxchg16b (dte, &old, &ldte.raw128[0 ]); + res =cmpxchg16b (dte, &old, &ldte.raw128[0 ]); -/* -* Hardware does not update the DTE behind our backs, so the -* return value should match "old". -*/ -if ( res != old ) - { -printk (XENLOG_ERR -"Dom%d: unexpected DTE %016lx_%016lx (expected %016lx_%016lx)\n" , - domain_id, - (uint64_t )(res >>64 ), (uint64_t )res, - (uint64_t )(old >>64 ), (uint64_t )old); - ret = -EILSEQ ; - } - } -else /* Best effort, updating domain_id last. */ +/* +* Hardware does not update the DTE behind our backs, so the +* return value should match "old". +*/ +if ( res != old ) { -uint64_t *ptr = (void *)dte; - -write_atomic (ptr +0 , ldte.raw64[0 ]); -/* No barrier should be needed between these two. */ -write_atomic (ptr +1 , ldte.raw64[1 ]); - - ret =1 ; +printk (XENLOG_ERR +"Dom%d: unexpected DTE %016lx_%016lx (expected %016lx_%016lx)\n" , + domain_id, + (uint64_t )(res >>64 ), (uint64_t )res, + (uint64_t )(old >>64 ), (uint64_t )old); + ret = -EILSEQ ; }return ret;
xen/drivers/passthrough/amd/pci_amd_iommu.c +6 -0
@@ -312,6 +312,12 @@ static int __init cf_check iov_detect(void)return -ENODEV ; } +if (unlikely (!cpu_has_cx16) ) + { +printk ("AMD-Vi: CPU doesn't support CMPXCHG16B, disabling\n" ); +return -ENOSYS ; + } + init_done =1 ;if ( !amd_iommu_perdev_intremap )
Signature · 9 lines
2.44.0 Teddy Astie | Vates XCP-ng Intern XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
PATCH v2 3/3 VT-d: Cleanup MAP_SINGLE_DEVICE and related code
In reply to Teddy AstieThis flag was only used in case cx16 is not available, as those code paths no longer exist, this flag now does basically nothing.
--- xen/drivers/passthrough/vtd/iommu.c | 12 +++--------- xen/drivers/passthrough/vtd/vtd.h | 5 ++--- 2 files changed, 5 insertions(+), 12 deletions(-)
xen/drivers/passthrough/vtd/iommu.c +3 -9
@@ -1692,15 +1692,9 @@ static int domain_context_mapping(struct domain *domain, u8 devfn,break ; } -if ( domain != pdev->domain && pdev->domain != dom_io ) - { -if ( pdev->domain->is_dying ) - mode |=MAP_OWNER_DYING ; -else if ( drhd && - !any_pdev_behind_iommu (pdev->domain, pdev, drhd->iommu) && - !pdev->phantom_stride ) - mode |=MAP_SINGLE_DEVICE ; - } +if ( domain != pdev->domain && pdev->domain != dom_io && + pdev->domain->is_dying ) + mode |=MAP_OWNER_DYING ;switch ( pdev->type ) {
xen/drivers/passthrough/vtd/vtd.h +2 -3
@@ -28,9 +28,8 @@*/ #define MAP_WITH_RMRR (1u << 0)#define MAP_OWNER_DYING (1u << 1) -#define MAP_SINGLE_DEVICE (1u << 2) -#define MAP_ERROR_RECOVERY (1u << 3) -#define UNMAP_ME_PHANTOM_FUNC (1u << 4) +#define MAP_ERROR_RECOVERY (1u << 2) +#define UNMAP_ME_PHANTOM_FUNC (1u << 3)/* Allow for both IOAPIC and IOSAPIC. */ #define IO_xAPIC_route_entry IO_APIC_route_entry
Signature · 9 lines
2.44.0 Teddy Astie | Vates XCP-ng Intern XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
Quoted 23 lines · Teddy Astie
All hardware that supports VT-d/AMD-Vi that exists also supports cx16 (aside specifically crafted virtual machines).
Some IOMMU code paths in Xen consider cases where VT-d/AMD-Vi is supported while cx16 isn't, those paths may be bugged and are barely tested, dead code in practice.
Disable IOMMU in case we have IOMMU hardware but no cx16, then cleanup no-cx16 handling logic from VT-d and AMD-Vi drivers.
Teddy
Changed in v2:
* Added cleanup no-cx16 code for x2APIC
* Fixed commit and code formatting
* Added missing Suggested-by noteTeddy Astie (3):
VT-d: Disable IOMMU if cx16 isn't supported
AMD-Vi: Disable IOMMU if cx16 isn't supported
VT-d: CleanupMAP_SINGLE_DEVICEand related code
Quoted 8 lines
xen/arch/x86/apic.c | 6 ++ xen/drivers/passthrough/amd/iommu_map.c | 42 ++++------ xen/drivers/passthrough/amd/pci_amd_iommu.c | 6 ++ xen/drivers/passthrough/vtd/intremap.c | 65 ++++----------- xen/drivers/passthrough/vtd/iommu.c | 92 +++++++-------------- xen/drivers/passthrough/vtd/vtd.h | 5 +- 6 files changed, 71 insertions(+), 145 deletions(-)
Sorry, but you've sent out two copies of each patch in this series, and it's not clear if they're identical or not.
Please could you send out another version, making sure there's only one of each patch.
Also, you need to swap ENOSYS with ENODEV, as per Jan's review on v1.
Thanks,
~Andrew
Quoted 2 lines · Andrew Cooper
Sorry, but you've sent out two copies of each patch in this series, and it's not clear if they're identical or not.
FWIW I've got just one copy.
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Le 11/04/2024 à 22:05, Andrew Cooper a écrit :
Quoted 36 lines · Teddy Astie
Quoted 23 lines
All hardware that supports VT-d/AMD-Vi that exists also supports cx16 (aside specifically crafted virtual machines).
Some IOMMU code paths in Xen consider cases where VT-d/AMD-Vi is supported while cx16 isn't, those paths may be bugged and are barely tested, dead code in practice.
Disable IOMMU in case we have IOMMU hardware but no cx16, then cleanup no-cx16 handling logic from VT-d and AMD-Vi drivers.
Teddy
Changed in v2:
* Added cleanup no-cx16 code for x2APIC
* Fixed commit and code formatting
* Added missing Suggested-by noteTeddy Astie (3):
VT-d: Disable IOMMU if cx16 isn't supported
AMD-Vi: Disable IOMMU if cx16 isn't supported
VT-d: CleanupMAP_SINGLE_DEVICEand related codeQuoted 8 lines
xen/arch/x86/apic.c | 6 ++ xen/drivers/passthrough/amd/iommu_map.c | 42 ++++------ xen/drivers/passthrough/amd/pci_amd_iommu.c | 6 ++ xen/drivers/passthrough/vtd/intremap.c | 65 ++++----------- xen/drivers/passthrough/vtd/iommu.c | 92 +++++++-------------- xen/drivers/passthrough/vtd/vtd.h | 5 +- 6 files changed, 71 insertions(+), 145 deletions(-)Sorry, but you've sent out two copies of each patch in this series, and it's not clear if they're identical or not.
Quoted 8 lines
Please could you send out another version, making sure there's only one of each patch.
Also, you need to swap ENOSYS with ENODEV, as per Jan's review on v1.
Thanks,
~Andrew
Hello,
Not entirely sure why it got sent twice, as marek said he only received it once. Will double-check next time to avoid this issue in case I wrongfully sent it twice.
Will also swap ENOSYS with ENODEV in the next version.
Thanks,
Teddy
Teddy Astie | Vates XCP-ng Intern
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Quoted 41 lines · Teddy Astie
Le 11/04/2024 à 22:05, Andrew Cooper a écrit :
Quoted 40 lines · Teddy Astie
Quoted 23 lines
All hardware that supports VT-d/AMD-Vi that exists also supports cx16 (aside specifically crafted virtual machines).
Some IOMMU code paths in Xen consider cases where VT-d/AMD-Vi is supported while cx16 isn't, those paths may be bugged and are barely tested, dead code in practice.
Disable IOMMU in case we have IOMMU hardware but no cx16, then cleanup no-cx16 handling logic from VT-d and AMD-Vi drivers.
Teddy
Changed in v2:
* Added cleanup no-cx16 code for x2APIC
* Fixed commit and code formatting
* Added missing Suggested-by noteTeddy Astie (3):
VT-d: Disable IOMMU if cx16 isn't supported
AMD-Vi: Disable IOMMU if cx16 isn't supported
VT-d: CleanupMAP_SINGLE_DEVICEand related codeQuoted 8 lines
xen/arch/x86/apic.c | 6 ++ xen/drivers/passthrough/amd/iommu_map.c | 42 ++++------ xen/drivers/passthrough/amd/pci_amd_iommu.c | 6 ++ xen/drivers/passthrough/vtd/intremap.c | 65 ++++----------- xen/drivers/passthrough/vtd/iommu.c | 92 +++++++-------------- xen/drivers/passthrough/vtd/vtd.h | 5 +- 6 files changed, 71 insertions(+), 145 deletions(-)Sorry, but you've sent out two copies of each patch in this series, and it's not clear if they're identical or not.
Please could you send out another version, making sure there's only one of each patch.
Also, you need to swap ENOSYS with ENODEV, as per Jan's review on v1.
Quoted 8 lines
Quoted 3 lines
Thanks,
~Andrew
Hello,
Not entirely sure why it got sent twice, as marek said he only received it once. Will double-check next time to avoid this issue in case I wrongfully sent it twice.
Huh, lore agrees. I seem to have both a direct and list copy which weren't correctly deduplicated.
Sorry for the noise.
Quoted 1 line
Will also swap ENOSYS with ENODEV in the next version.
Thanks.
~Andrew