iommu/amd-vi: adjust _amd_iommu_flush_pages() to handle pseudo-domids
PATCH 1/1 iommu/amd-vi: adjust _amd_iommu_flush_pages() to handle pseudo-domids
When the passed domain is DomIO iterate over the list of DomIO assigned devices and flush each pseudo-domid found.
invalidate_all_domain_pages() does call amd_iommu_flush_all_pages() with DomIO as a parameter, and hence the underlying _amd_iommu_flush_pages() implementation must be capable of flushing all pseudo-domids used by the quarantine domain logic.
While there fix invalidate_all_domain_pages() to only attempt to flush the domains that have IOMMU enabled, otherwise the flush is pointless.
---
Only build tested, as I don't have an AMD system that I can do suspend/resume on.
--- xen/drivers/passthrough/amd/iommu_cmd.c | 29 ++++++++++++++++++++---- xen/drivers/passthrough/amd/iommu_init.c | 4 +++- 2 files changed, 28 insertions(+), 5 deletions(-)
xen/drivers/passthrough/amd/iommu_cmd.c +25 -4
@@ -330,13 +330,34 @@ static void _amd_iommu_flush_pages(struct domain *d, daddr_t daddr, unsigned int order) {struct amd_iommu *iommu; -unsigned int dom_id = d->domain_id;/* send INVALIDATE_IOMMU_PAGES command */ -for_each_amd_iommu ( iommu ) +if ( d != dom_io ) { -invalidate_iommu_pages (iommu, daddr, dom_id, order); -flush_command_buffer (iommu,0 ); +for_each_amd_iommu ( iommu ) + { +invalidate_iommu_pages (iommu, daddr, d->domain_id, order); +flush_command_buffer (iommu,0 ); + } + } +else + { +const struct pci_dev *pdev; + +for_each_pdev (dom_io, pdev) +if ( pdev->arch.pseudo_domid !=DOMID_INVALID ) + { + iommu =find_iommu_for_device (pdev->sbdf.seg, pdev->sbdf.bdf); +if ( !iommu ) + { +ASSERT_UNREACHABLE (); +continue ; + } + +invalidate_iommu_pages (iommu, daddr, pdev->arch.pseudo_domid, + order); +flush_command_buffer (iommu,0 ); + } }if ( ats_enabled )
xen/drivers/passthrough/amd/iommu_init.c +3 -1
@@ -1532,8 +1532,10 @@ int __init amd_iommu_init_late(void)static void invalidate_all_domain_pages (void ) {struct domain *d; +for_each_domain ( d ) -amd_iommu_flush_all_pages (d); +if (is_iommu_enabled (d) ) +amd_iommu_flush_all_pages (d); } static int cf_check _invalidate_all_devices(
Signature · 1 line
2.40.0
Quoted 5 lines · Roger Pau Monne
When the passed domain is DomIO iterate over the list of DomIO assigned devices and flush each pseudo-domid found.
invalidate_all_domain_pages()does callamd_iommu_flush_all_pages()with DomIO as a parameter,
Does it? Since the full function is visible in the patch (because of the "While there ..." change), it seems pretty clear that it doesn't:for_each_domain() iterates over ordinary domains only.
Quoted 3 lines
and hence the underlying
_amd_iommu_flush_pages()implementation must be capable of flushing all pseudo-domids used by the quarantine domain logic.
While it didn't occur to me right away when we discussed this, it may well be that I left alone all flushing when introducing the pseudo domain IDs simply because no flushing would ever happen for the quarantine domain.
Quoted 2 lines
While there fix
invalidate_all_domain_pages()to only attempt to flush the domains that have IOMMU enabled, otherwise the flush is pointless.
For the moment at least it looks to me as if this change alone wants to go in.
Jan
Quoted 10 lines · Jan Beulich
Quoted 5 lines · Roger Pau Monne
When the passed domain is DomIO iterate over the list of DomIO assigned devices and flush each pseudo-domid found.
invalidate_all_domain_pages()does callamd_iommu_flush_all_pages()with DomIO as a parameter,Does it? Since the full function is visible in the patch (because of the "While there ..." change), it seems pretty clear that it doesn't:
for_each_domain()iterates over ordinary domains only.
Oh, I got confused by domain_create() returning early for system domains.
Quoted 8 lines
Quoted 3 lines
and hence the underlying
_amd_iommu_flush_pages()implementation must be capable of flushing all pseudo-domids used by the quarantine domain logic.While it didn't occur to me right away when we discussed this, it may well be that I left alone all flushing when introducing the pseudo domain IDs simply because no flushing would ever happen for the quarantine domain.
But the purpose of the calls to invalidate_all_devices() and invalidate_all_domain_pages() in amd_iommu_resume() is to cover up for the lack of Invalidate All support in the IOMMU, so flushing pseudo-domids is also required in order to flush all possible IOMMU state.
Note that as part of invalidate_all_devices() we do invalidate DTEs for devices assigned to pseudo-domids, hence it seems natural that we also flush such pseudo-domids.
Quoted 5 lines
Quoted 2 lines
While there fix
invalidate_all_domain_pages()to only attempt to flush the domains that have IOMMU enabled, otherwise the flush is pointless.For the moment at least it looks to me as if this change alone wants to go in.
I would rather get the current patch with an added call to flush dom_io in invalidate_all_domain_pages().
Thanks, Roger.
Quoted 34 lines · Roger Pau Monné
Quoted 10 lines · Jan Beulich
Quoted 5 lines · Roger Pau Monne
When the passed domain is DomIO iterate over the list of DomIO assigned devices and flush each pseudo-domid found.
invalidate_all_domain_pages()does callamd_iommu_flush_all_pages()with DomIO as a parameter,Does it? Since the full function is visible in the patch (because of the "While there ..." change), it seems pretty clear that it doesn't:
for_each_domain()iterates over ordinary domains only.Oh, I got confused by
domain_create()returning early for system domains.Quoted 8 lines
Quoted 3 lines
and hence the underlying
_amd_iommu_flush_pages()implementation must be capable of flushing all pseudo-domids used by the quarantine domain logic.While it didn't occur to me right away when we discussed this, it may well be that I left alone all flushing when introducing the pseudo domain IDs simply because no flushing would ever happen for the quarantine domain.
But the purpose of the calls to
invalidate_all_devices()andinvalidate_all_domain_pages()inamd_iommu_resume()is to cover up for the lack of Invalidate All support in the IOMMU, so flushing pseudo-domids is also required in order to flush all possible IOMMU state.Note that as part of
invalidate_all_devices()we do invalidate DTEs for devices assigned to pseudo-domids, hence it seems natural that we also flush such pseudo-domids.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
While there fix
invalidate_all_domain_pages()to only attempt to flush the domains that have IOMMU enabled, otherwise the flush is pointless.For the moment at least it looks to me as if this change alone wants to go in.
I would rather get the current patch with an added call to flush
dom_ioininvalidate_all_domain_pages().
The question is: Is there anything that needs flushing for the quarantine domain. Right now I'm thinking that there isn't.
Jan