powerpc: pSeries: vfio: iommu: Re-enable support for SPAPR TCE VFIO
Subsystems (5)
v4 cover powerpc: pSeries: vfio: iommu: Re-enable support for SPAPR TCE VFIO
The patches reimplement the iommu table_group_ops for pSeries for VFIO SPAPR TCE sub-driver thereby bringing consistency with PowerNV implementation and getting rid of limitations/bugs which were emanating from these differences on the earlier approach on pSeries.
Structure of the patchset:
-------------------------
The first and fifth patches just code movements.
Second patch takes care of collecting the TCE and DDW information for the vfio_iommu_spapr_tce_ddw_info during probe.
Third patch fixes the convention of using table[1] for VFs on pSeries when used by the host driver.
Fourth patch fixes the VFIO to call TCE clear before unset window.
The last patch has the API reimplementations, please find the details on its commit description.
Testing:
-------
Tested with nested guest for NVME card, Mellanox multi-function card by attaching them to nested kvm guest running on a pSeries lpar.
Also vfio-test [2] by Alex Willamson, was forked and updated to add support for pSeries guest and used to test these patches[3].
Limitations/Known Issues:
------------------------
* The DMA window restrictions with SRIOV VF scenarios of having maximum 1 dma window is taken care in the current patches itself. However, the necessary changes required in vfio_iommu_spapr_tce_ddw_info to expose the default window being a 64-bit one and the qemu changes handle the same will be taken care in future versions.
References:
----------
[1] https://lore.kernel.org/linuxppc-dev/[email protected]/
[2] https://github.com/awilliam/tests
[3] https://github.com/nnmwebmin/vfio-ppc-tests/tree/vfio-ppc-ex
---
Changelog:
v3: https://lore.kernel.org/linuxppc-dev/[email protected]/
- Rebased to top of the tree.
- In the first patch, spapr_tce_table_group_ops is only used locally in
the pseries/iommu.c file. So, made it static.
- Fixed the test robot reported issues [email protected]
and [email protected]
- Updated the commit messages
v2: https://lore.kernel.org/linuxppc-dev/[email protected]/
- Rebased to upstream. So, required the explicit vmalloc.h inclusion as its
removed from the system header io.h now.
- Fixed the DLPAR hotplugged device assignment case. The dma window
property is backed up before removal. That copy is restored when required.
- Cleaned up bit more. Removed leftover debug prints and dump_stack()s.
- The warning at remap_pfn_range_notrack() during kvm guest boot is no longer
seen after the rebase.
v1: https://lore.kernel.org/linuxppc-dev/[email protected]/
- Rewrite as to stop borrowing the DMA windows and implemented
the table_group_ops for pSeries.
- Cover letter and Patch 6 has more details as this was a rewrite.
Shivaprasad G Bhat (6):
powerpc/iommu: Move pSeries specific functions to pseries/iommu.c powerpc/pseries/iommu: Fix the VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl output powerpc/pseries/iommu: Use the iommu table[0] for IOV VF's DDW vfio/spapr: Always clear TCEs before unsetting the window powerpc/iommu: Move dev_has_iommu_table() to iommu.c powerpc/iommu: Reimplement the iommu_table_group_ops for pSeries
arch/powerpc/include/asm/iommu.h | 14 +- arch/powerpc/kernel/eeh.c | 16 - arch/powerpc/kernel/iommu.c | 170 +----- arch/powerpc/platforms/powernv/pci-ioda.c | 6 +- arch/powerpc/platforms/pseries/iommu.c | 704 +++++++++++++++++++++- drivers/vfio/vfio_iommu_spapr_tce.c | 13 +- 6 files changed, 718 insertions(+), 205 deletions(-)
--
Signature
PATCH v4 1/6 powerpc/iommu: Move pSeries specific functions to pseries/iommu.c
The PowerNV specific table_group_ops are defined in powernv/pci-ioda.c. The pSeries specific table_group_ops are sitting in the generic powerpc file. Move it to where it actually belong(pseries/iommu.c).
The functions are currently defined even for CONFIG_PPC_POWERNV which are unused on PowerNV.
Only code movement, no functional changes intended.
--- arch/powerpc/include/asm/iommu.h | 4 + arch/powerpc/kernel/iommu.c | 149 -------------------------------- arch/powerpc/platforms/pseries/iommu.c | 144 +++++++++++++++++++++++++++++++ 3 files changed, 149 insertions(+), 148 deletions(-)
arch/powerpc/include/asm/iommu.h +3 -1
@@ -156,6 +156,9 @@ extern int iommu_tce_table_put(struct iommu_table *tbl);extern struct iommu_table *iommu_init_table (struct iommu_table *tbl,int nid,unsigned long res_start,unsigned long res_end);bool iommu_table_in_use (struct iommu_table *tbl); +extern void iommu_table_reserve_pages (struct iommu_table *tbl, +unsigned long res_start,unsigned long res_end); +extern void iommu_table_clear (struct iommu_table *tbl);#define IOMMU_TABLE_GROUP_MAX_TABLES 2
@@ -218,7 +221,6 @@ extern long iommu_tce_xchg_no_kill(struct mm_struct *mm,extern void iommu_tce_kill (struct iommu_table *tbl,unsigned long entry,unsigned long pages); -extern struct iommu_table_group_ops spapr_tce_table_group_ops;#else static inline void iommu_register_group(struct iommu_table_group *table_group, int pci_domain_number,
arch/powerpc/kernel/iommu.c +2 -147
173 lines not on this page · raw message
arch/powerpc/platforms/pseries/iommu.c +144 -0
162 lines not on this page · raw message
PATCH v4 2/6 powerpc/pseries/iommu: Fix the VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl output
In reply to Shivaprasad G BhatThe ioctl VFIO_IOMMU_SPAPR_TCE_GET_INFO is not reporting the actuals on the platform as not all the details are correctly collected during the platform probe/scan into the iommu_table_group.
Collect the information during the device setup time as the DMA window property is already looked up on parent nodes anyway.
--- arch/powerpc/platforms/pseries/iommu.c | 81 ++++++++++++++++++++++++++------ 1 file changed, 67 insertions(+), 14 deletions(-)
arch/powerpc/platforms/pseries/iommu.c +67 -14
@@ -865,13 +865,6 @@ static void pci_dma_bus_setup_pSeriesLP(struct pci_bus *bus)be32_to_cpu (prop.tce_shift),NULL , &iommu_table_lpar_multi_ops); -/* Only for normal boot with default window. Doesn't matter even -* if we set these with DDW which is 64bit during kdump, since -* these will not be used during kdump. -*/ - ppci->table_group->tce32_start =be64_to_cpu (prop.dma_base); - ppci->table_group->tce32_size =1 <<be32_to_cpu (prop.window_shift); -if (!iommu_init_table (tbl, ppci->phb->node,0 ,0 ))panic ("Failed to initialize iommu table" );
@@ -1623,6 +1616,71 @@ static bool enable_ddw(struct pci_dev *dev, struct device_node *pdn)return direct_mapping; } +static __u64 query_page_size_to_mask (u32 query_page_size) +{ +const long shift[] = { + (SZ_4K ), (SZ_64K ), (SZ_16M ), + (SZ_32M ), (SZ_64M ), (SZ_128M ), + (SZ_256M ), (SZ_16G ), (SZ_2M ) + }; +int i, ret =0 ; + +for (i =0 ; i <ARRAY_SIZE (shift); i++) { +if (query_page_size & (1 << i)) + ret |= shift[i]; + } + +return ret; +} + +static void spapr_tce_init_table_group (struct pci_dev *pdev, +struct device_node *pdn, +struct dynamic_dma_window_prop prop) +{ +struct iommu_table_group *table_group =PCI_DN (pdn)->table_group; +u32 ddw_avail[DDW_APPLICABLE_SIZE ]; + +struct ddw_query_response query; +int ret; + +/* Only for normal boot with default window. Doesn't matter during +* kdump, since these will not be used during kdump. +*/ +if (is_kdump_kernel ()) +return ; + +if (table_group->max_dynamic_windows_supported !=0 ) +return ;/* already initialized */ + + table_group->tce32_start =be64_to_cpu (prop.dma_base); + table_group->tce32_size =1 <<be32_to_cpu (prop.window_shift); + +if (!of_find_property (pdn,"ibm,dma-window" ,NULL )) +dev_err (&pdev->dev,"default dma window missing!\n" ); + + ret =of_property_read_u32_array (pdn,"ibm,ddw-applicable" , + &ddw_avail[0 ],DDW_APPLICABLE_SIZE ); +if (ret) { + table_group->max_dynamic_windows_supported =-1 ; +return ; + } + + ret =query_ddw (pdev, ddw_avail, &query, pdn); +if (ret) { +dev_err (&pdev->dev,"%s: query_ddw failed\n" , __func__); + table_group->max_dynamic_windows_supported =-1 ; +return ; + } + +if (query.windows_available ==0 ) + table_group->max_dynamic_windows_supported =1 ; +else + table_group->max_dynamic_windows_supported =IOMMU_TABLE_GROUP_MAX_TABLES ; + + table_group->max_levels =1 ; + table_group->pgsizes |=query_page_size_to_mask (query.page_size); +} +static void pci_dma_dev_setup_pSeriesLP (struct pci_dev *dev) {struct device_node *pdn, *dn;
@@ -1662,13 +1720,6 @@ static void pci_dma_dev_setup_pSeriesLP(struct pci_dev *dev)be32_to_cpu (prop.tce_shift),NULL , &iommu_table_lpar_multi_ops); -/* Only for normal boot with default window. Doesn't matter even -* if we set these with DDW which is 64bit during kdump, since -* these will not be used during kdump. -*/ - pci->table_group->tce32_start =be64_to_cpu (prop.dma_base); - pci->table_group->tce32_size =1 <<be32_to_cpu (prop.window_shift); -iommu_init_table (tbl, pci->phb->node,0 ,0 );iommu_register_group (pci->table_group,pci_domain_nr (pci->phb->bus),0 );
@@ -1677,6 +1728,8 @@ static void pci_dma_dev_setup_pSeriesLP(struct pci_dev *dev)pr_debug (" found DMA window, table: %p\n" , pci->table_group); } +spapr_tce_init_table_group (dev, pdn, prop); +set_iommu_table_base (&dev->dev, pci->table_group->tables[0 ]);iommu_add_device (pci->table_group, &dev->dev); }
PATCH v4 3/6 powerpc/pseries/iommu: Use the iommu table[0] for IOV VF's DDW
In reply to Shivaprasad G BhatThis patch basically brings consistency with PowerNV approach to use the first freely available iommu table when the default window is removed.
The pSeries iommu code convention has been that the table[0] is for the default 32 bit DMA window and the table[1] is for the 64 bit DDW.
With VFs having only 1 DMA window, the default has to be removed for creating the larger DMA window. The existing code uses the table[1] for that, while marking the table[0] as NULL. This is fine as long as the host driver itself uses the device.
For the VFIO user, on pSeries there is no way to skip table[0] as the VFIO subdriver uses the first freely available table. The window 0, when created as 64-bit DDW in that context would still be on table[0], as the maximum number of windows is 1.
This won't have any impact for the host driver as the table is fetched from the device's iommu_table_base.
--- arch/powerpc/platforms/pseries/iommu.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
arch/powerpc/platforms/pseries/iommu.c +7 -5
@@ -155,7 +155,7 @@ static void iommu_pseries_free_group(struct iommu_table_group *table_group,#endif /* Default DMA window table is at index 0, while DDW at 1. SR-IOV -* adapters only have table on index 1. +* adapters only have table on index 0(if not direct mapped). */ if (table_group->tables[0 ])iommu_tce_table_put (table_group->tables[0 ]);
@@ -1527,6 +1527,11 @@ static bool enable_ddw(struct pci_dev *dev, struct device_node *pdn)clean_dma_window (pdn, win64->value); goto out_del_list; } +if (default_win_removed) { +iommu_tce_table_put (pci->table_group->tables[0 ]); + pci->table_group->tables[0 ] =NULL ; +set_iommu_table_base (&dev->dev,NULL ); + } }else {struct iommu_table *newtbl;int i;
@@ -1556,15 +1561,12 @@ static bool enable_ddw(struct pci_dev *dev, struct device_node *pdn) 1UL << len, page_shift, NULL, &iommu_table_lpar_multi_ops);iommu_init_table (newtbl, pci->phb->node, start, end); - pci->table_group->tables[1 ] = newtbl; + pci->table_group->tables[default_win_removed ?0 :1 ] = newtbl;set_iommu_table_base (&dev->dev, newtbl); }if (default_win_removed) { -iommu_tce_table_put (pci->table_group->tables[0 ]); - pci->table_group->tables[0 ] =NULL ; -/* default_win is valid here because default_win_removed == true */ of_remove_property (pdn, default_win);dev_info (&dev->dev,"Removed default DMA window for %pOF\n" , pdn);
PATCH v4 4/6 vfio/spapr: Always clear TCEs before unsetting the window
In reply to Shivaprasad G BhatThe PAPR expects the TCE table to have no entries at the time of unset window(i.e. remove-pe). The TCE clear right now is done before freeing the iommu table. On pSeries, the unset window makes those entries inaccessible to the OS and the H_PUT/GET calls fail on them with H_CONSTRAINED.
On PowerNV, this has no side effect as the TCE clear can be done before the DMA window removal as well.
--- drivers/vfio/vfio_iommu_spapr_tce.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
drivers/vfio/vfio_iommu_spapr_tce.c +9 -4
@@ -364,7 +364,6 @@ static void tce_iommu_release(void *iommu_data)if (!tbl)continue ; -tce_iommu_clear (container, tbl, tbl->it_offset, tbl->it_size);tce_iommu_free_table (container, tbl); }
@@ -720,6 +719,8 @@ static long tce_iommu_remove_window(struct tce_container *container,BUG_ON (!tbl->it_size); +tce_iommu_clear (container, tbl, tbl->it_offset, tbl->it_size); +/* Detach groups from IOMMUs */ list_for_each_entry (tcegrp, &container->group_list, next) { table_group =iommu_group_get_iommudata (tcegrp->grp);
@@ -738,7 +739,6 @@ static long tce_iommu_remove_window(struct tce_container *container, }/* Free table */ -tce_iommu_clear (container, tbl, tbl->it_offset, tbl->it_size);tce_iommu_free_table (container, tbl); container->tables[num] =NULL ;
@@ -1197,9 +1197,14 @@ static void tce_iommu_release_ownership(struct tce_container *container,return ; } -for (i =0 ; i <IOMMU_TABLE_GROUP_MAX_TABLES ; ++i) -if (container->tables[i]) +for (i =0 ; i <IOMMU_TABLE_GROUP_MAX_TABLES ; ++i) { +if (container->tables[i]) { +tce_iommu_clear (container, container->tables[i], + container->tables[i]->it_offset, + container->tables[i]->it_size); table_group->ops->unset_window (table_group, i); + } + } } static long tce_iommu_take_ownership(struct tce_container *container,
PATCH v4 5/6 powerpc/iommu: Move dev_has_iommu_table() to iommu.c
In reply to Shivaprasad G BhatMove function dev_has_iommu_table() to powerpc/kernel/iommu.c as it is going to be used by machine specific iommu code as well in subsequent patches.
--- arch/powerpc/include/asm/iommu.h | 6 ++++++ arch/powerpc/kernel/eeh.c | 16 ---------------- arch/powerpc/kernel/iommu.c | 17 +++++++++++++++++ 3 files changed, 23 insertions(+), 16 deletions(-)
arch/powerpc/include/asm/iommu.h +6 -0
@@ -220,6 +220,7 @@ extern long iommu_tce_xchg_no_kill(struct mm_struct *mm, enum dma_data_direction *direction);extern void iommu_tce_kill (struct iommu_table *tbl,unsigned long entry,unsigned long pages); +int dev_has_iommu_table (struct device *dev,void *data);#else static inline void iommu_register_group(struct iommu_table_group *table_group,
@@ -233,6 +234,11 @@ static inline int iommu_add_device(struct iommu_table_group *table_group, {return 0 ; } + +static inline int dev_has_iommu_table (struct device *dev,void *data) +{ +return 0 ; +}#endif /* !CONFIG_IOMMU_API */ u64 dma_iommu_get_required_mask (struct device *dev);
arch/powerpc/kernel/eeh.c +0 -16
@@ -1273,22 +1273,6 @@ EXPORT_SYMBOL(eeh_dev_release);#ifdef CONFIG_IOMMU_API -static int dev_has_iommu_table (struct device *dev,void *data) -{ -struct pci_dev *pdev =to_pci_dev (dev); -struct pci_dev **ppdev = data; - -if (!dev) -return 0 ; - -if (device_iommu_mapped (dev)) { - *ppdev = pdev; -return 1 ; - } - -return 0 ; -} -/** * eeh_iommu_group_to_pe - Convert IOMMU group to EEH PE * @group: IOMMU group
arch/powerpc/kernel/iommu.c +17 -0
@@ -988,6 +988,23 @@ unsigned long iommu_direction_to_tce_perm(enum dma_data_direction dir)EXPORT_SYMBOL_GPL (iommu_direction_to_tce_perm);#ifdef CONFIG_IOMMU_API + +int dev_has_iommu_table (struct device *dev,void *data) +{ +struct pci_dev *pdev =to_pci_dev (dev); +struct pci_dev **ppdev = data; + +if (!dev) +return 0 ; + +if (device_iommu_mapped (dev)) { + *ppdev = pdev; +return 1 ; + } + +return 0 ; +} +/* * SPAPR TCE API */
PATCH v4 6/6 powerpc/iommu: Reimplement the iommu_table_group_ops for pSeries
In reply to Shivaprasad G BhatPPC64 IOMMU API defines iommu_table_group_ops which handles DMA windows for PEs, their ownership transfer, create/set/unset the TCE tables for the Dynamic DMA wundows(DDW). VFIOS uses these APIs for support on POWER.
The commit 9d67c9433509 ("powerpc/iommu: Add "borrowing" iommu_table_group_ops") implemented partial support for this API with "borrow" mechanism wherein the DMA windows if created already by the host driver, they would be available for VFIO to use. Also, it didn't have the support to control/modify the window size or the IO page size.
The current patch implements all the necessary iommu_table_group_ops APIs there by avoiding the "borrrowing". So, just the way it is on the PowerNV platform, with this patch the iommu table group ownership is transferred to the VFIO PPC subdriver, the iommu table, DMA windows creation/deletion all driven through the APIs.
The pSeries uses the query-pe-dma-window, create-pe-dma-window and reset-pe-dma-window RTAS calls for DMA window creation, deletion and reset to defaul. The RTAs calls do show some minor differences to the way things are to be handled on the pSeries which are listed below.
* On pSeries, the default DMA window size is "fixed" cannot be custom sized as requested by the user. For non-SRIOV VFs, It is fixed at 2GB and for SRIOV VFs, its variable sized based on the capacity assigned to it during the VF assignment to the LPAR. So, for the default DMA window alone the size if requested less than tce32_size, the smaller size is enforced using the iommu table->it_size.
* The DMA start address for 32-bit window is 0, and for the 64-bit window in case of PowerNV is hardcoded to TVE select (bit 59) at 512PiB offset. This address is returned at the time of create_table() API call (even before the window is created), the subsequent set_window() call actually opens the DMA window. On pSeries, the DMA start address for 32-bit window is known from the 'ibm,dma-window' DT property. However, the 64-bit window start address is not known until the create-pe-dma RTAS call is made. So, the create_table() which returns the DMA window start address actually opens the DMA window and returns the DMA start address as returned by the Hypervisor for the create-pe-dma RTAS call.
* The reset-pe-dma RTAS call resets the DMA windows and restores the default DMA window, however it does not clear the TCE table entries if there are any. In case of ownership transfer from platform domain which used direct mapping, the patch chooses remove-pe-dma instead of reset-pe for the 64-bit window intentionally so that the clear_dma_window() is called.
Other than the DMA window management changes mentioned above, the patch also brings back the userspace view for the single level TCE as it existed before commit 090bad39b237a ("powerpc/powernv: Add indirect levels to it_userspace") along with the relavent refactoring.
--- arch/powerpc/include/asm/iommu.h | 4 arch/powerpc/kernel/iommu.c | 4 arch/powerpc/platforms/powernv/pci-ioda.c | 6 arch/powerpc/platforms/pseries/iommu.c | 627 +++++++++++++++++++++++++---- 4 files changed, 543 insertions(+), 98 deletions(-)
arch/powerpc/include/asm/iommu.h +2 -2
@@ -181,9 +181,9 @@ struct iommu_table_group_ops {long (*unset_window)(struct iommu_table_group *table_group,int num);/* Switch ownership from platform code to external user (e.g. VFIO) */ -long (*take_ownership)(struct iommu_table_group *table_group); +long (*take_ownership)(struct iommu_table_group *table_group,struct device *dev);/* Switch ownership from external user (e.g. VFIO) back to core */ -void (*release_ownership)(struct iommu_table_group *table_group); +void (*release_ownership)(struct iommu_table_group *table_group,struct device *dev); };struct iommu_table_group_link {
arch/powerpc/kernel/iommu.c +2 -2
@@ -1171,7 +1171,7 @@ spapr_tce_platform_iommu_attach_dev(struct iommu_domain *platform_domain,* The domain being set to PLATFORM from earlier * BLOCKED. The table_group ownership has to be released. */ - table_group->ops->release_ownership (table_group); + table_group->ops->release_ownership (table_group, dev);iommu_group_put (grp);return 0 ;
@@ -1199,7 +1199,7 @@ spapr_tce_blocked_iommu_attach_dev(struct iommu_domain *platform_domain,* also sets the dma_api ops */ table_group =iommu_group_get_iommudata (grp); - ret = table_group->ops->take_ownership (table_group); + ret = table_group->ops->take_ownership (table_group, dev);iommu_group_put (grp);return ret;
arch/powerpc/platforms/powernv/pci-ioda.c +4 -2
@@ -1537,7 +1537,8 @@ static void pnv_ioda_setup_bus_dma(struct pnv_ioda_pe *pe, struct pci_bus *bus) } } -static long pnv_ioda2_take_ownership (struct iommu_table_group *table_group) +static long pnv_ioda2_take_ownership (struct iommu_table_group *table_group, +struct device *dev__maybe_unused ) { struct pnv_ioda_pe *pe = container_of(table_group, struct pnv_ioda_pe, table_group);
@@ -1562,7 +1563,8 @@ static long pnv_ioda2_take_ownership(struct iommu_table_group *table_group)return 0 ; } -static void pnv_ioda2_release_ownership (struct iommu_table_group *table_group) +static void pnv_ioda2_release_ownership (struct iommu_table_group *table_group, +struct device *dev__maybe_unused ) { struct pnv_ioda_pe *pe = container_of(table_group, struct pnv_ioda_pe, table_group);
arch/powerpc/platforms/pseries/iommu.c +535 -92
761 lines not on this page · raw message
Michael Ellerman
AppliedQuoted 11 lines · Shivaprasad G Bhat
The patches reimplement the iommu
table_group_opsfor pSeries for VFIO SPAPR TCE sub-driver thereby bringing consistency with PowerNV implementation and getting rid of limitations/bugs which were emanating from these differences on the earlier approach on pSeries.Structure of the patchset:
-------------------------
The first and fifth patches just code movements.[...]
Applied to powerpc/next.
[1/6] powerpc/iommu: Move pSeries specific functions to pseries/iommu.c
https://git.kernel.org/powerpc/c/b09c031d9433dda3186190e5845ba0d720212567 [2/6] powerpc/pseries/iommu: Fix the VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl output https://git.kernel.org/powerpc/c/6af67f2ddfcbbca551d786415beda14c48fb6ddf [3/6] powerpc/pseries/iommu: Use the iommu table[0] for IOV VF's DDW https://git.kernel.org/powerpc/c/aed6e4946ed9654fc965482b045b84f9b9572bb8 [4/6] vfio/spapr: Always clear TCEs before unsetting the window https://git.kernel.org/powerpc/c/4ba2fdff2eb174114786784926d0efb6903c88a6 [5/6] powerpc/iommu: Move dev_has_iommu_table() to iommu.c https://git.kernel.org/powerpc/c/35146eadcb81d72153a1621f3cc0d5588cae19d3 [6/6] powerpc/iommu: Reimplement the iommu_table_group_ops for pSeries https://git.kernel.org/powerpc/c/f431a8cde7f102fce412546db6e62fdbde1131a7
cheers