iommu/arm: Add Renesas IPMMU-VMSA support
PATCH 1/1 iommu/arm: Add Renesas IPMMU-VMSA support
From: Oleksandr Tyshchenko <[email protected]>
The purpose of this patch is to add IPMMU-VMSA support to Xen on ARM.
The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.
Please note, this driver is supposed to work only with newest Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).
This driver is based on Linux's IPMMU-VMSA driver from Renesas BSP:
https://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git/tree/drivers/iommu/ipmmu-vmsa.c?h=v4.14.75-ltsi/rcar-3.9.2
and Xen's SMMU driver:xen/drivers/passthrough/arm/smmu.c
Although Xen driver has a lot in common with Linux driver, it is not a "direct ported" copy and should be treated as such.
Driver was tested on Gen3 H3 ES3.0 based boards using current staging (7d1460c xen/arm: optee: fix compilation with GCC 4.8)
in a system with several DMA masters being assigned to different guest domains.
You can find it here:
repo: https://github.com/otyshchenko1/xen.git branch: ipmmu_upstream1
Oleksandr Tyshchenko (1):
iommu/arm: Add Renesas IPMMU-VMSA support
xen/arch/arm/platforms/Kconfig | 1 + xen/drivers/passthrough/Kconfig | 13 + xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/ipmmu-vmsa.c | 1487 ++++++++++++++++++++++++++++++ 4 files changed, 1502 insertions(+) create mode 100644 xen/drivers/passthrough/arm/ipmmu-vmsa.c
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 1/1 iommu/arm: Add Renesas IPMMU-VMSA support
From: Oleksandr Tyshchenko <[email protected]>
The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.
Please note, current driver is supposed to work only with newest Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).
--- xen/arch/arm/platforms/Kconfig | 1 + xen/drivers/passthrough/Kconfig | 13 + xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/ipmmu-vmsa.c | 1487 ++++++++++++++++++++++++++++++ 4 files changed, 1502 insertions(+) create mode 100644 xen/drivers/passthrough/arm/ipmmu-vmsa.c
xen/arch/arm/platforms/Kconfig +1 -0
@@ -25,6 +25,7 @@ config RCAR3 bool "Renesas RCar3 support" depends on ARM_64 select HAS_SCIF + select IPMMU_VMSA ---help--- Enable all the required drivers for Renesas RCar3
xen/drivers/passthrough/Kconfig +13 -0
@@ -12,4 +12,17 @@ config ARM_SMMU Say Y here if your SoC includes an IOMMU device implementing the ARM SMMU architecture. + +config IPMMU_VMSA + bool "Renesas IPMMU-VMSA found in R-Car Gen3 SoCs" + default y + depends on ARM_64 + ---help--- + Support for implementations of the Renesas IPMMU-VMSA found + in R-Car Gen3 SoCs. + + Say Y here if you are using newest R-Car Gen3 SoCs revisions which IPMMU + hardware supports stage 2 translation table format and is able to use + CPU's P2M table as is. + endif
xen/drivers/passthrough/arm/Makefile +1 -0
@@ -1,2 +1,3 @@ obj-y += iommu.o obj-$(CONFIG_ARM_SMMU) += smmu.o +obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +1487 -0
1,487 lines not on this page · raw message
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Hi,
Apologies for the late answer. I have been traveling for the past few weeks.
Quoted 5 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.
Do you have a link to the specification?
My knowledge about the IPMMU is quite limited, so for now, I will mostly look at Xen specific code. It would be good if someone with a better knowledge of the driver could have a look.
[...]
xen/drivers/passthrough/Kconfig config ARM_SMMU · line 18
Quoted 12 lines
xen/drivers/passthrough/Kconfig +4
-0@@ -12,4 +12,17 @@ config ARM_SMMU Say Y here if your SoC includes an IOMMU device implementing the ARM SMMU architecture. + +config IPMMU_VMSA + bool "Renesas IPMMU-VMSA found in R-Car Gen3 SoCs" + default y
I would prefer this to be a default N for the time-being.
xen/drivers/passthrough/Kconfig config ARM_SMMU · line 24
Quoted 6 lines
xen/drivers/passthrough/Kconfig +6
-0+ depends on ARM_64 + ---help--- + Support for implementations of the Renesas IPMMU-VMSA found + in R-Car Gen3 SoCs. + + Say Y here if you are using newest R-Car Gen3 SoCs revisions which IPMMU
What new now will be old soon ;). So it would be best if you give a precise revision here.
Quoted 23 lines
xen/drivers/passthrough/Kconfig +3
-0+ hardware supports stage 2 translation table format and is able to use + CPU's P2M table as is. + endifxen/drivers/passthrough/arm/Makefile +1
-0@@ -1,2 +1,3 @@ obj-y += iommu.o obj-$(CONFIG_ARM_SMMU) += smmu.o +obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.oxen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0@@ -0,0 +1,1487 @@ +/* +* xen/drivers/passthrough/arm/ipmmu-vmsa.c +* +* Driver for the Renesas IPMMU-VMSA found in R-Car Gen3 SoCs. +*
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 15
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+* The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) +* which provides address translation and access protection functionalities +* to processing units and interconnect networks. +* +* Please note, current driver is supposed to work only with newest Gen3 SoCs +* revisions which IPMMU hardware supports stage 2 translation table format and +* is able to use CPU's P2M table as is. +* +* Based on Linux's IPMMU-VMSA driver from Renesas BSP: +* drivers/iommu/ipmmu-vmsa.c
What are the major differences compare the Linux driver?
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 19
Quoted 4 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +4
-0+* you can found at: +* url: git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git +* branch: v4.14.75-ltsi/rcar-3.9.2 +* commit: a5266d298124874c2c06b8b13d073f6ecc2ee355
Is there any reason to use the BSP driver and not the one provided by Linux directly?
[...]
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 106
Quoted 5 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0+#define dev_archdata (dev) ((struct ipmmu_vmsa_xen_device *)dev->archdata.iommu) + +/* Root/Cache IPMMU device's information */ +struct ipmmu_vmsa_device +{
AFAICT, this file was converted to Xen coding style. If so, the style for struct is
struct ... {
...
};
Quoted 12 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +12
-0+struct device *dev; +void __iomem *base; +struct ipmmu_vmsa_device *root; +struct list_head list; +unsigned int num_utlbs; +unsigned int num_ctx; +spinlock_t lock;/* Protects ctx and domains[] */ +DECLARE_BITMAP (ctx,IPMMU_CTX_MAX ); +struct ipmmu_vmsa_domain *domains[IPMMU_CTX_MAX ]; +}; + +/*
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 128
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+* Root/Cache IPMMU domain's information +* +* Root IPMMU device is assigned to Root IPMMU domain while Cache IPMMU device +* is assigned to Cache IPMMU domain. Master devices are connected to Cache +* IPMMU devices through specific ports called micro-TLBs. +* All Cache IPMMU devices, in turn, are connected to Root IPMMU device +* which manages IPMMU context. +*/ +struct ipmmu_vmsa_domain +{
Ditto.
Quoted 12 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +12
-0+/* +* IPMMU device assigned to this IPMMU domain. +* Either Root device which is located at the main memory bus domain or +* Cache device which is located at each hierarchy bus domain. +*/ +struct ipmmu_vmsa_device *mmu; + +/* Context used for this IPMMU domain */ +unsigned int context_id; + +/* Xen domain associated with this IPMMU domain */ +struct domain *d;
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 150
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ +/* The fields below are used for Cache IPMMU domain only */ + +/* +* Used to keep track of the master devices which are attached to this +* IPMMU domain (domain users). Master devices behind the same IPMMU device +* are grouped together by putting into the same IPMMU domain. +* Only when the refcount reaches 0 this IPMMU domain can be destroyed. +*/ +int refcount;
If the refcount cannot be 0, then I would prefer an unsigned int here.
Quoted 3 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +3
-0+/* Used to link this IPMMU domain for the same Xen domain */ +struct list_head list; +};
[...]
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 316
Quoted 2 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +2
-0+/* Read/Write Access */ +static u32 ipmmu_read (struct ipmmu_vmsa_device *mmu,unsigned int offset)
If you are going to use Xen coding style, then this should be "uint32_t". The same is valid everywhere in this file, I am not going to mention all of them :).
Quoted 18 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +18
-0+{ +return readl (mmu->base + offset); +} + +static void ipmmu_write (struct ipmmu_vmsa_device *mmu,unsigned int offset, +u32 data) +{ +writel (data, mmu->base + offset); +} + +static u32 ipmmu_ctx_read_root (struct ipmmu_vmsa_domain *domain, +unsigned int reg) +{ +return ipmmu_read (domain->mmu->root, + domain->context_id *IM_CTX_SIZE + reg); +} + +static void ipmmu_ctx_write_root(struct ipmmu_vmsa_domain *domain,
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 344
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ unsigned int reg, u32 data) +{ +ipmmu_write (domain->mmu->root, + domain->context_id *IM_CTX_SIZE + reg, data); +} + +static void ipmmu_ctx_write_cache (struct ipmmu_vmsa_domain *domain, +unsigned int reg,u32 data) +{ +ASSERT (reg ==IMCTR );
What's the rationale of passing reg in parameter if it can only be equal to IMCTR?
Quoted 6 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+ +/* Mask fields which are implemented in IPMMU-MM only. */ +if ( !ipmmu_is_root (domain->mmu) ) +ipmmu_write (domain->mmu, domain->context_id *IM_CTX_SIZE + reg, + data &IMCTR_COMMON_MASK ); +}
[...]
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 407
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+/* Enable MMU translation for the micro-TLB. */ +static void ipmmu_utlb_enable (struct ipmmu_vmsa_domain *domain, +unsigned int utlb) +{ +struct ipmmu_vmsa_device *mmu = domain->mmu; + +/* +* TODO: Reference-count the micro-TLB as several bus masters can be +* connected to the same micro-TLB. +*/
What would be the exact problem if this is not handled? Could a utlb shared between multiple domain?
Quoted 4 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +4
-0+ipmmu_write (mmu,IMUASID (utlb),0 ); +ipmmu_write (mmu,IMUCTR (utlb),ipmmu_read (mmu,IMUCTR (utlb)) | +IMUCTR_TTSEL_MMU (domain->context_id) |IMUCTR_MMUEN ); +}
[...]
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 460
Quoted 3 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +3
-0+static int ipmmu_domain_init_context (struct ipmmu_vmsa_domain *domain) +{ +u64 ttbr;
s/u64/uint64_t/
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8
-0+u32 tsz0; +int ret; + +/* Find an unused context. */ + ret =ipmmu_domain_allocate_context (domain->mmu->root, domain); +if ( ret <0 ) +return ret; +
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 478
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ domain->context_id = ret; + +/* +* TTBR0 +* Use P2M table for this Xen domain. +*/ +ASSERT (domain->d !=NULL ); + ttbr =page_to_maddr (domain->d->arch.p2m.root); + + dev_info(domain->mmu->root->dev, "d%d: Set IPMMU context %u (pgd 0x%"PRIx64")\n",
We introduce a format specifier to print a domain. So you can use %pd in combination of just domain->d.
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 489
Quoted 11 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +11
-0+ domain->d->domain_id, domain->context_id, ttbr); + +ipmmu_ctx_write_root (domain,IMTTLBR0 , ttbr &IMTTLBR0_TTBR_MASK ); +ipmmu_ctx_write_root (domain,IMTTUBR0 , (ttbr >>32 ) &IMTTUBR0_TTBR_MASK ); + +/* +* TTBCR +* We use long descriptors with inner-shareable WBWA tables and allocate +* the whole "p2m_ipa_bits" IPA space to TTBR0. Use 4KB page granule. +* Start page table walks at first level. Bypass stage 1 translation +* when only stage 2 translation is performed.
I am not sure to understand the last sentence. You only use stage2 right, so it shouldn't it be "Always bypass stage 1 translation"?
Quoted 5 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0+*/ + tsz0 = (64 - p2m_ipa_bits) <<IMTTBCR_TSZ0_SHIFT ; +ipmmu_ctx_write_root (domain,IMTTBCR ,IMTTBCR_EAE |IMTTBCR_PMB | +IMTTBCR_SH0_INNER_SHAREABLE |IMTTBCR_ORGN0_WB_WA | +IMTTBCR_IRGN0_WB_WA |IMTTBCR_SL0_LVL_1 | tsz0);
[...]
Quoted 16 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +16
-0+/* Fault Handling */ +static void ipmmu_domain_irq (struct ipmmu_vmsa_domain *domain) +{ +const u32 err_mask =IMSTR_MHIT |IMSTR_ABORT |IMSTR_PF |IMSTR_TF ; +struct ipmmu_vmsa_device *mmu = domain->mmu; +u32 status; +u64 iova; + + status =ipmmu_ctx_read_root (domain,IMSTR ); +if ( !(status & err_mask) ) +return ; + + iova =ipmmu_ctx_read_root (domain,IMELAR ) | + ((u64 )ipmmu_ctx_read_root (domain,IMEUAR ) <<32 ); + +/*
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 556
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+* Clear the error status flags. Unlike traditional interrupt flag +* registers that must be cleared by writing 1, this status register +* seems to require 0. The error address register must be read before, +* otherwise its value will be 0. +*/ +ipmmu_ctx_write_root (domain,IMSTR ,0 ); + +/* Log fatal errors. */ +if ( status &IMSTR_MHIT ) + dev_err_ratelimited(mmu->dev, "d%d: Multiple TLB hits @0x%"PRIx64"\n",
Similar remark for d%d here ...
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 559
Quoted 3 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +3
-0+ domain->d->domain_id, iova); +if ( status &IMSTR_ABORT ) + dev_err_ratelimited(mmu->dev, "d%d: Page Table Walk Abort @0x%"PRIx64"\n",
... here and ...
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 569
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ domain->d->domain_id, iova); + +/* Return if it is neither Permission Fault nor Translation Fault. */ +if ( !(status & (IMSTR_PF |IMSTR_TF )) ) +return ; + +/* Flush the TLB as required when IPMMU translation error occurred. */ +ipmmu_tlb_invalidate (domain); + + dev_err_ratelimited(mmu->dev, "d%d: Unhandled fault: status 0x%08x iova 0x%"PRIx64"\n",
... here.
Quoted 31 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +31
-0+ domain->d->domain_id, status, iova); +} + +static void ipmmu_irq (int irq,void *dev,struct cpu_user_regs *regs) +{ +struct ipmmu_vmsa_device *mmu = dev; +unsigned int i; +unsigned long flags; + +spin_lock_irqsave (&mmu->lock, flags); + +/* +* When interrupt arrives, we don't know the context it is related to. +* So, check interrupts for all active contexts to locate a context +* with status bits set. +*/ +for ( i =0 ; i < mmu->num_ctx; i++ ) + { +if ( !mmu->domains[i] ) +continue ; +ipmmu_domain_irq (mmu->domains[i]); + } + +spin_unlock_irqrestore (&mmu->lock, flags); +} + +/* Master devices management */ +static int ipmmu_attach_device (struct ipmmu_vmsa_domain *domain, +struct device *dev) +{ +struct ipmmu_vmsa_master_cfg *cfg =dev_archdata (dev)->cfg;
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 610
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+struct ipmmu_vmsa_device *mmu = cfg->mmu; +unsigned int i; + +if ( !mmu ) + { +dev_err (dev,"Cannot attach to IPMMU\n" ); +return -ENXIO ; + } + +if ( !domain->mmu )
So you read domain->mmu here and ...
Quoted 6 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+ { +/* The domain hasn't been used yet, initialize it. */ + domain->mmu = mmu; + +/* +* We have already enabled context for Root IPMMU assigned to this
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 626
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+* Xen domain in ipmmu_domain_init_context(). +* Enable the context for Cache IPMMU only. Flush the TLB as required +* when modifying the context registers. +*/ +ipmmu_ctx_write_cache (domain,IMCTR , +ipmmu_ctx_read_root (domain,IMCTR ) |IMCTR_FLUSH ); + +dev_info (dev,"Using IPMMU context %u\n" , domain->context_id); + } +else if ( domain->mmu != mmu )
... here. What actually promise that domain->mmu can't change in parallel?
Quoted 17 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +17
-0+ { +/* +* Something is wrong, we can't attach two master devices using +* different IOMMUs to the same IPMMU domain. +*/ +dev_err (dev,"Can't attach IPMMU %s to domain on IPMMU %s\n" , +dev_name (mmu->dev),dev_name (domain->mmu->dev)); +return -EINVAL ; + } + else + dev_info(dev, "Reusing IPMMU context %u\n", domain->context_id); + +for ( i =0 ; i < cfg->num_utlbs; ++i ) +ipmmu_utlb_enable (domain, cfg->utlbs[i]); + +return 0 ; +}
[...]
Quoted 5 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0+static void ipmmu_protect_masters (struct ipmmu_vmsa_device *mmu) +{ +struct dt_device_node *node; + +dt_for_each_device_node ( dt_host, node )
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 769
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ { +if ( mmu->dev->of_node !=dt_parse_phandle (node,"iommus" ,0 ) ) +continue ; + +/* Let Xen know that the master device is protected by an IOMMU. */ +dt_device_set_protected (node); + +dev_info (mmu->dev,"Found master device %s\n" ,dt_node_full_name (node)); + } +}
I don't much like this. You are going to go through the whole DT quite a few times.
The IOMMU interface in Xen has not been designed with the new IOMMU bindings in mind. I would prefer if we look for extending add_device callback to support platform device.
This would allow to probe the device later on and therefore avoid to go through the device-tree multiple.
Quoted 5 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0+ +static void ipmmu_device_reset (struct ipmmu_vmsa_device *mmu) +{ +unsigned int i; +
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 784
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+/* Disable all contexts. */ +for ( i =0 ; i < mmu->num_ctx; ++i ) +ipmmu_write (mmu, i *IM_CTX_SIZE +IMCTR ,0 ); +} + +/* +* This function relies on the fact that Root IPMMU device is being probed +* the first. If not the case, it denies further Cache IPMMU device probes +* (returns the -ENODEV) until the Root IPMMU device has been registered +* for sure.
Can we look at handling -EDEFER in Xen instead?
Quoted 1 line
+ */
[...]
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1235
Quoted 3 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +3
-0+static int __must_check ipmmu_map_page (struct domain *d,dfn_t dfn,mfn_t mfn, +unsigned int flags, +unsigned int *flush_flags)
The function is exactly the same as for the SMMU driver. Could we implement common helpers in a separate file?
Quoted 19 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +19
-0+{ +p2m_type_t t; + +/* +* Grant mappings can be used for DMA requests. The dev_bus_addr +* returned by the hypercall is the MFN (not the IPA). For device +* protected by an IOMMU, Xen needs to add a 1:1 mapping in the domain +* p2m to allow DMA request to work. +* This is only valid when the domain is directed mapped. Hence this +* function should only be used by gnttab code with gfn == mfn == dfn. +*/ +BUG_ON (!is_domain_direct_mapped (d)); +BUG_ON (mfn_x (mfn) !=dfn_x (dfn)); + +/* We only support readable and writable flags */ +if ( !(flags & (IOMMUF_readable | IOMMUF_writable)) ) +return -EINVAL ; + + t = (flags & IOMMUF_writable) ? p2m_iommu_map_rw : p2m_iommu_map_ro;
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1264
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ +/* +* The function guest_physmap_add_entry replaces the current mapping +* if there is already one... +*/ +return guest_physmap_add_entry (d,_gfn (dfn_x (dfn)),_mfn (dfn_x (dfn)),0 , t); +} + +static int __must_check ipmmu_unmap_page (struct domain *d,dfn_t dfn, +unsigned int *flush_flags)
Same here.
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1274
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+{ +/* +* This function should only be used by gnttab code when the domain +* is direct mapped (i.e. gfn == mfn == dfn). +*/ +if ( !is_domain_direct_mapped (d) ) +return -EINVAL ; + +return guest_physmap_remove_page (d,_gfn (dfn_x (dfn)),_mfn (dfn_x (dfn)),0 ); +}
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 1 line · Julien Grall
Hi,
Hi, Julien.
Quoted 2 lines
Apologies for the late answer. I have been traveling for the past few weeks.
Absolutely no problem. Thank you for your review.
Quoted 9 lines · Oleksandr Tyshchenko
Quoted 5 lines
From: Oleksandr Tyshchenko <[email protected]>
The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.
Do you have a link to the specification?
All I have is a TRM. Unfortunately, I can't share it.
Quoted 2 lines
My knowledge about the IPMMU is quite limited, so for now, I will mostly look at Xen specific code. It would be good if someone with a better knowledge of the driver could have a look.
I understand your point. Make sense. I will CC a person who is familiar with H/W.
Quoted 4 lines
[...]
Quoted 13 lines
xen/drivers/passthrough/Kconfig config ARM_SMMU · line 18
Quoted 11 lines
xen/drivers/passthrough/Kconfig +0
-0@@ -12,4 +12,17 @@ config ARM_SMMUSay Y here if your SoC includes an IOMMU device implementing the ARM SMMU architecture.
xen/drivers/passthrough/Kconfig +4
-0+ +config IPMMU_VMSA + bool "Renesas IPMMU-VMSA found in R-Car Gen3 SoCs" + default yI would prefer this to be a default N for the time-being.
ok
Quoted 11 lines
Quoted 9 lines
+ depends on
ARM_64
+ ---help---
+ Support for implementations of the Renesas IPMMU-VMSA found + in R-Car Gen3 SoCs.
+
+ Say Y here if you are using newest R-Car Gen3 SoCs revisions which IPMMUWhat new now will be old soon ;). So it would be best if you give a precise revision here.
yes ;). Will clarify.
Quoted 33 lines
Quoted 33 lines
+ hardware supports stage 2 translation table format and is able to use
+ CPU's P2M table as is.
+
endifxen/drivers/passthrough/arm/Makefile +0
-0@@ -1,2 +1,3 @@obj-y += iommu.o
obj-$(CONFIG_ARM_SMMU) += smmu.oxen/drivers/passthrough/arm/Makefile +1
-0+obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.oxen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7
-0@@ -0,0 +1,1487 @@ +/* +* xen/drivers/passthrough/arm/ipmmu-vmsa.c +* +* Driver for the Renesas IPMMU-VMSA found in R-Car Gen3 SoCs. +* +* The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) +* which provides address translation and access protection functionalities
+ * to processing units and interconnect networks.
+ *
+ * Please note, current driver is supposed to work only with newest Gen3 SoCs
Quoted 8 lines
Quoted 6 lines
+ * revisions which IPMMU hardware supports stage 2 translation table format and
+ * is able to use CPU's P2M table as is.
+ *
+ * Based on Linux's IPMMU-VMSA driver from Renesas BSP:
+ *drivers/iommu/ipmmu-vmsa.cWhat are the major differences compare the Linux driver?
Well, the major differences are:
1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the page table with the CPU. Stage 1 translation is always bypassed in Xen driver.
So, Xen driver is supposed to be used with newest Gen3 SoC revisions only (H3 ES3.0, M3 ES3.0, etc.) which IPMMU hardware does support stage 2 translation table format.
2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.
3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.
Quoted 9 lines
Quoted 7 lines
+ * you can found at:
+ * url:git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git
+ * branch: v4.14.75-ltsi/rcar-3.9.2
+ * commit:a5266d298124874c2c06b8b13d073f6ecc2ee355Is there any reason to use the BSP driver and not the one provided by Linux directly?
I was thinking the BSP driver is a *little bit* more updated than Linux one. Sometime it was a big difference between mainline and BSP driver. But now
the difference is not big and mostly in DDR_BACKUP and WHITELIST support. I looked at mainline driver as well when implementing Xen driver.
Quoted 13 lines
[...]
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 106
Quoted 6 lines
+#define
dev_archdata(dev)((structipmmu_vmsa_xen_device*)dev->archdata.iommu)xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +4
-0+ +/* Root/Cache IPMMU device's information */ +struct ipmmu_vmsa_device +{AFAICT, this file was converted to Xen coding style. If so, the style for struct is
Quoted 3 lines
struct ... {
...
};
Yes, will correct everywhere in this file.
Quoted 19 lines
Quoted 19 lines
+ struct device *dev;
+ void__iomem*base;
+ structipmmu_vmsa_device*root;
+ structlist_headlist;
+ unsigned intnum_utlbs;
+ unsigned intnum_ctx;
+spinlock_tlock; /* Protects ctx and domains[] */ +DECLARE_BITMAP(ctx, IPMMU_CTX_MAX);
+ structipmmu_vmsa_domain*domains[IPMMU_CTX_MAX];xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+}; + +/* +* Root/Cache IPMMU domain's information +* +* Root IPMMU device is assigned to Root IPMMU domain while Cache IPMMU device
+ * is assigned to Cache IPMMU domain. Master devices are connected to Cache
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c
Quoted 6 lines
+ * IPMMU devices through specific ports called micro-TLBs.
+ * All Cache IPMMU devices, in turn, are connected to Root IPMMU device
+ * which manages IPMMU context.
+ */xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +2
-0+struct ipmmu_vmsa_domain +{Ditto.
ok
Quoted 21 lines
Quoted 21 lines
+ /*
+ * IPMMU device assigned to this IPMMU domain.
+ * Either Root device which is located at the main memory bus domain or
+ * Cache device which is located at each hierarchy bus domain. + */
+ structipmmu_vmsa_device*mmu;xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +11
-0+ + /* Context used for this IPMMU domain */ + unsigned int context_id; + + /* Xen domain associated with this IPMMU domain */ + struct domain *d; + + /* The fields below are used for Cache IPMMU domain only */ + + /* +* Used to keep track of the master devices which are attached to this
+ * IPMMU domain (domain users). Master devices behind the same
Quoted 8 lines
Quoted 6 lines
IPMMU device
+ * are grouped together by putting into the same IPMMU domain. + * Only when the refcount reaches 0 this IPMMU domain can be destroyed.
+ */
+ int refcount;If the refcount cannot be 0, then I would prefer an unsigned int here.
It can be >= 0.
Quoted 6 lines
Quoted 4 lines
+ /* Used to link this IPMMU domain for the same Xen domain */ + struct
list_headlist;
+};
Quoted 7 lines
[...]
Quoted 3 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +2
-0+/* Read/Write Access */ +static u32 ipmmu_read(struct ipmmu_vmsa_device *mmu, unsigned intoffset)
If you are going to use Xen coding style, then this should be "
uint32_t". The same is valid everywhere in this file, I am not going to mention all of them :).
Yes, will change everywhere in this file.
Quoted 24 lines
Quoted 24 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0+{ + return readl(mmu->base + offset); +} + +static void ipmmu_write(struct ipmmu_vmsa_device *mmu, unsigned intoffset,
+ u32 data)xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +16
-0+{ + writel(data, mmu->base + offset); +} + +static u32 ipmmu_ctx_read_root (struct ipmmu_vmsa_domain *domain, + unsigned int reg) +{ + return ipmmu_read(domain->mmu->root, + domain->context_id * IM_CTX_SIZE + reg); +} + +static void ipmmu_ctx_write_root (struct ipmmu_vmsa_domain *domain, + unsigned int reg, u32 data) +{ + ipmmu_write(domain->mmu->root, + domain->context_id * IM_CTX_SIZE + reg, data);
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 344
Quoted 6 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+} + +static void ipmmu_ctx_write_cache (struct ipmmu_vmsa_domain *domain, + unsigned int reg, u32 data) +{ + ASSERT(reg == IMCTR);What's the rationale of passing reg in parameter if it can only be equal to IMCTR?
Good question. I tried to retain the same interface as for ipmmu_ctx_write_root(_all) for visibility.
Cache IPMMU device has other than IMCTR context registers, but they are not used by this driver.
Shall I drop reg parameter?
Quoted 11 lines
Quoted 8 lines
+
+ /* Mask fields which are implemented in IPMMU-MM only. */ + if ( !ipmmu_is_root(domain->mmu) )
+ ipmmu_write(domain->mmu,domain->context_id*IM_CTX_SIZE+ reg,
+ data &IMCTR_COMMON_MASK);
+}[...]
Quoted 13 lines
Quoted 11 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8
-0+/* Enable MMU translation for the micro-TLB. */ +static void ipmmu_utlb_enable (struct ipmmu_vmsa_domain *domain, + unsigned int utlb) +{ + struct ipmmu_vmsa_device *mmu = domain->mmu; + + /* +* TODO: Reference-count the micro-TLB as several bus masters can be
+ * connected to the same micro-TLB.
+ */What would be the exact problem if this is not handled? Could a utlb shared between multiple domain?
Actually, this TODO as well as the whole "TLB and micro-TLB Management" code I tried to retain as much as possible.
The large amount of devices have unique micro-TLB connection, but there is a case, when not. At the moment, I don't see any problem if these potential devices (which share the same utlb) are assigned to the same Xen domain.
In this case we would just enable the same utlb more than once at the domain creation time (assign a device) and disable the same utlb more than once at the domain destruction time (deassign a device).
The worst case scenario would be when these devices are assigned to different Xen domains. So, I think, the same utlb *can't* be shared between multiple Xen domains, since it points to the context bank to use for the page walk.
Quoted 7 lines
Quoted 6 lines
+
ipmmu_write(mmu, IMUASID(utlb), 0);
+ ipmmu_write(mmu, IMUCTR(utlb),ipmmu_read(mmu, IMUCTR(utlb))| +IMUCTR_TTSEL_MMU(domain->context_id)|IMUCTR_MMUEN); +}
Quoted 7 lines
[...]
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 460
Quoted 3 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +3
-0+static int ipmmu_domain_init_context (struct ipmmu_vmsa_domain *domain) +{ + u64 ttbr;s/u64/uint64_t/
ok
Quoted 14 lines
Quoted 14 lines
+ u32 tsz0;
+ int ret;xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +11
-0+ + /* Find an unused context. */ + ret = ipmmu_domain_allocate_context(domain->mmu->root, domain); + if ( ret < 0 ) + return ret; + + domain->context_id = ret; + + /* +* TTBR0 +* Use P2M table for this Xen domain.
Quoted 8 lines
Quoted 6 lines
+ */
+ASSERT(domain->d != NULL);
+ ttbr =page_to_maddr(domain->d->arch.p2m.root);
+
+ dev_info(domain->mmu->root->dev, "d%d: Set IPMMU context %u (pgd 0x%"PRIx64")\n",We introduce a format specifier to print a domain. So you can use %pd in combination of just
domain->d.
Such a convenient thing. Will use.
Quoted 10 lines
Quoted 10 lines
+
domain->d->domain_id,domain->context_id, ttbr); +
+ipmmu_ctx_write_root(domain, IMTTLBR0, ttbr & IMTTLBR0_TTBR_MASK); +ipmmu_ctx_write_root(domain, IMTTUBR0, (ttbr >> 32) & IMTTUBR0_TTBR_MASK);
+
+ /*
+ * TTBCR
+ * We use long descriptors with inner-shareable WBWA tables and
Quoted 8 lines
Quoted 6 lines
allocate
+ * the whole "p2m_ipa_bits" IPA space to TTBR0. Use 4KB page granule.
+ * Start page table walks at first level. Bypass stage 1 translation
+ * when only stage 2 translation is performed.I am not sure to understand the last sentence. You only use stage2 right, so it shouldn't it be "Always bypass stage 1 translation"?
Yes, you are right. Will rephrase.
Quoted 35 lines
Quoted 8 lines
+ */
+ tsz0 = (64 -p2m_ipa_bits) <<IMTTBCR_TSZ0_SHIFT;
+ ipmmu_ctx_write_root(domain, IMTTBCR,IMTTBCR_EAE|IMTTBCR_PMB| +IMTTBCR_SH0_INNER_SHAREABLE|IMTTBCR_ORGN0_WB_WA|
+IMTTBCR_IRGN0_WB_WA|IMTTBCR_SL0_LVL_1| tsz0);[...]
Quoted 24 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +4
-0+/* Fault Handling */ +static void ipmmu_domain_irq (struct ipmmu_vmsa_domain *domain) +{ + const u32 err_mask = IMSTR_MHIT | IMSTR_ABORT | IMSTR_PF |
IMSTR_TF;
+ structipmmu_vmsa_device*mmu =domain->mmu;
+ u32 status;
+ u64 iova;xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +11
-0+ + status = ipmmu_ctx_read_root(domain, IMSTR); + if ( !(status & err_mask) ) + return; + + iova = ipmmu_ctx_read_root(domain, IMELAR) | + ((u64 )ipmmu_ctx_read_root (domain,IMEUAR ) <<32 ); + + /* +* Clear the error status flags. Unlike traditional interrupt flag +* registers that must be cleared by writing 1, this status register
+ * seems to require 0. The error address register must be read before,
+ * otherwise its value will be 0.
+ */
Quoted 8 lines
Quoted 6 lines
+
ipmmu_ctx_write_root(domain, IMSTR, 0);
+
+ /* Log fatal errors. */
+ if ( status &IMSTR_MHIT)
+ dev_err_ratelimited(mmu->dev, "d%d: Multiple TLB hits @0x%"PRIx64"\n",Similar remark for d%d here ...
ok
Quoted 7 lines
Quoted 5 lines
+
domain->d->domain_id, iova);
+ if ( status &IMSTR_ABORT)
+ dev_err_ratelimited(mmu->dev, "d%d: Page Table Walk Abort @0x%"PRIx64"\n",... here and ...
ok
Quoted 7 lines
Quoted 7 lines
+
domain->d->domain_id, iova);
+
+ /* Return if it is neither Permission Fault nor Translation Fault. */
+ if ( !(status & (IMSTR_PF|IMSTR_TF)) )
+ return;
+
Quoted 8 lines
Quoted 6 lines
+ /* Flush the TLB as required when IPMMU translation error occurred. */
+ipmmu_tlb_invalidate(domain);
+
+ dev_err_ratelimited(mmu->dev, "d%d: Unhandled fault: status 0x%08x iova 0x%"PRIx64"\n",... here.
ok
Quoted 59 lines
Quoted 33 lines
+
domain->d->domain_id, status, iova);xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +12
-0+} + +static void ipmmu_irq (int irq,void *dev,struct cpu_user_regs *regs) +{ + struct ipmmu_vmsa_device *mmu = dev; + unsigned int i; + unsigned long flags; + + spin_lock_irqsave(&mmu->lock, flags); + + /* +* When interrupt arrives, we don't know the context it is related to.
+ * So, check interrupts for all active contexts to locate a context + * with status bits set.
+ */
+ for ( i = 0; i <mmu->num_ctx; i++ )
+ {
+ if ( !mmu->domains[i] )
+ continue;
+ipmmu_domain_irq(mmu->domains[i]);
+ }xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +9
-0+ + spin_unlock_irqrestore(&mmu->lock, flags); +} + +/* Master devices management */ +static int ipmmu_attach_device (struct ipmmu_vmsa_domain *domain, + struct device *dev) +{ + struct ipmmu_vmsa_master_cfg *cfg = dev_archdata(dev)->cfg;xen/drivers/passthrough/arm/ipmmu-vmsa.c line 610
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ struct ipmmu_vmsa_device *mmu = cfg->mmu; + unsigned int i; + + if ( !mmu ) + { + dev_err(dev, "Cannot attach to IPMMU\n"); + return -ENXIO; + } + + if ( !domain->mmu )So you read
domain->mmuhere and ...Quoted 13 lines
+ {
+ /* The domain hasn't been used yet, initialize it. */ +domain->mmu= mmu;
+
+ /*
+ * We have already enabled context for Root IPMMU assigned to this
+ * Xen domain inipmmu_domain_init_context().
+ * Enable the context for Cache IPMMU only. Flush the TLB as required
+ * when modifying the context registers.
+ */
+ ipmmu_ctx_write_cache(domain, IMCTR,
Quoted 8 lines
Quoted 6 lines
+
ipmmu_ctx_read_root(domain, IMCTR)|IMCTR_FLUSH);
+
+dev_info(dev, "Using IPMMU context %u\n", domain->context_id); + }
+ else if (domain->mmu!= mmu )... here. What actually promise that
domain->mmucan't change in parallel?
ipmmu_attach_device is protected by xen_domain->lock
Quoted 33 lines
Quoted 19 lines
+ {
+ /*
+ * Something is wrong, we can't attach two master devices using + * different IOMMUs to the same IPMMU domain.
+ */
+ dev_err(dev, "Can't attach IPMMU %s to domain on IPMMU %s\n", +dev_name(mmu->dev),dev_name(domain->mmu->dev)); + return-EINVAL;
+ }
+ else
+dev_info(dev, "Reusing IPMMU context %u\n", domain->context_id);xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+ + for ( i = 0; i < cfg->num_utlbs; ++i ) + ipmmu_utlb_enable(domain, cfg->utlbs[i]); + + return 0; +}[...]
Quoted 11 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+static void ipmmu_protect_masters (struct ipmmu_vmsa_device *mmu) +{ + struct dt_device_node *node; + + dt_for_each_device_node( dt_host, node ) + { + if ( mmu->dev->of_node != dt_parse_phandle(node, "iommus", 0) ) + continue; + + /* Let Xen know that the master device is protected by anIOMMU. */
Quoted 8 lines
Quoted 6 lines
+
dt_device_set_protected(node);
+
+dev_info(mmu->dev, "Found master device %s\n", dt_node_full_name(node));
+ }
+}I don't much like this. You are going to go through the whole DT quite a few times.
+1
Quoted 4 lines
The IOMMU interface in Xen has not been designed with the new IOMMU bindings in mind. I would prefer if we look for extending
add_devicecallback to support platform device.This would allow to probe the device later on and therefore avoid to go through the device-tree multiple.
I completely agree with you that current implementation is not optimal and should be reworked in order not to scan the whole DT many times, but I am not completely understand what we should do and how exactly.
Could you, please, add more details?
Quoted 14 lines
Quoted 14 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +12
-0+ +static void ipmmu_device_reset (struct ipmmu_vmsa_device *mmu) +{ + unsigned int i; + + /* Disable all contexts. */ + for ( i = 0; i < mmu->num_ctx; ++i ) + ipmmu_write(mmu, i * IM_CTX_SIZE + IMCTR, 0); +} + +/* +* This function relies on the fact that Root IPMMU device is being
Quoted 8 lines
Quoted 6 lines
probed
+ * the first. If not the case, it denies further Cache IPMMU device probes
+ * (returns the-ENODEV) until the Root IPMMU device has been registered
+ * for sure.Can we look at handling
-EDEFERin Xen instead?
I am not sure this is something we should implement at this stage (while only IPMMU driver would be a user). I have already resolved that possible issue by trying to locate a Root IPMMU device and probe it the first
to avoid the case described above. So now, we don't depend on how IPMMU devices are located in DT. Please, see ipmmu_init(). So, I tend to live with it some time.
Quoted 11 lines
Quoted 2 lines
+ */
[...]
Quoted 4 lines
+static int
__must_checkipmmu_map_page(struct domain *d,dfn_tdfn,mfn_tmfn,
+ unsigned int flags, + unsigned int*flush_flags)The function is exactly the same as for the SMMU driver. Could we implement common helpers in a separate file?
Sounds indeed reasonable. I will look at it.
Quoted 31 lines
Quoted 31 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7
-0+{ + p2m_type_t t; + + /* +* Grant mappings can be used for DMA requests. The dev_bus_addr +* returned by the hypercall is the MFN (not the IPA). For device +* protected by an IOMMU, Xen needs to add a 1:1 mapping in the domain
+ * p2m to allow DMA request to work.
+ * This is only valid when the domain is directed mapped. Hence this
+ * function should only be used by gnttab code with gfn == mfn == dfn.
+ */
+ BUG_ON(!is_domain_direct_mapped(d));
+BUG_ON(mfn_x(mfn) != dfn_x(dfn));xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+ + /* We only support readable and writable flags */ + if ( !(flags & (IOMMUF_readable | IOMMUF_writable)) ) + return -EINVAL; + + t = (flags & IOMMUF_writable) ? p2m_iommu_map_rw :
p2m_iommu_map_ro;
+
+ /*
+ * The functionguest_physmap_add_entryreplaces the current mapping
+ * if there is already one...
+ */
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1264
Quoted 6 lines
+ return
guest_physmap_add_entry(d, _gfn(dfn_x(dfn)), _mfn(dfn_x(dfn)), 0, t);xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +4
-0+} + +static int __must_check ipmmu_unmap_page (struct domain *d,dfn_t dfn, + unsigned int *flush_flags)Same here.
ok
----------
Julien, what we should do with the fact that IPMMU supports only 3-level page table?
I left a TODO regarding that, but we need to work out some usable solution if possible.
/*
* As 4-level translation table is not supported in IPMMU, we need
* to check IPA size used for P2M table beforehand to be sure it is
* 3-level and the IPMMU will be able to use it.
*
* In case of using 4KB page granule we should use two concatenated
* translation tables at level 1 in order to support 40 bit IPA
* with 3-level translation table.
*
* TODO: Probably, when determing the "pa_range" in setup_virt_paging()
* we should take into the account the IPMMU ability as well. */
if ( IPMMU_MAX_P2M_IPA_BITS < p2m_ipa_bits )
{
dev_err(&node->dev, "P2M IPA size is not supported (P2M=%u IPMMU=%u)!\n", p2m_ipa_bits, IPMMU_MAX_P2M_IPA_BITS);
return -EOPNOTSUPP;
}
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Hi Oleksandr,
Quoted 10 lines · Oleksandr
Quoted 1 line · Julien Grall
Apologies for the late answer. I have been traveling for the past few weeks.
Absolutely no problem. Thank you for your review.
Quoted 4 lines · Oleksandr Tyshchenko
Quoted 2 lines
From: Oleksandr Tyshchenko <[email protected]>
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.
Do you have a link to the specification?
All I have is a TRM. Unfortunately, I can't share it.
Does anyone in the community has access to the spec?
Quoted 39 lines
Quoted 28 lines
Quoted 28 lines
+ hardware supports stage 2 translation table format and is able to use + CPU's P2M table as is.
+
endifxen/drivers/passthrough/arm/Makefile +0
-0@@ -1,2 +1,3 @@obj-y += iommu.o
obj-$(CONFIG_ARM_SMMU) += smmu.oxen/drivers/passthrough/arm/Makefile +1
-0+obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.oxen/drivers/passthrough/arm/ipmmu-vmsa.c new file +9
-0@@ -0,0 +1,1487 @@ +/* +* xen/drivers/passthrough/arm/ipmmu-vmsa.c +* +* Driver for the Renesas IPMMU-VMSA found in R-Car Gen3 SoCs. +* +* The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) +* which provides address translation and access protection functionalities +* to processing units and interconnect networks. +* Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 15
Quoted 6 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+* Please note, current driver is supposed to work only with newest Gen3 SoCs +* revisions which IPMMU hardware supports stage 2 translation table format and +* is able to use CPU's P2M table as is. +* +* Based on Linux's IPMMU-VMSA driver from Renesas BSP: +* drivers/iommu/ipmmu-vmsa.c What are the major differences compare the Linux driver?
Well, the major differences are:
Quoted 7 lines
1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the page table with the CPU. Stage 1 translation is always bypassed in Xen driver.
So, Xen driver is supposed to be used with newest Gen3 SoC revisions only (H3 ES3.0, M3 ES3.0, etc.) which IPMMU hardware does support stage 2 translation table format.
2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.
3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.
Can this be written in the commit message? This is helpful for anyone reviewing the driver today and future developer.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +4
-0+ * you can found at: + * url: git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git + * branch: v4.14.75-ltsi/rcar-3.9.2 + * commit: a5266d298124874c2c06b8b13d073f6ecc2ee355
Quoted 5 lines
Quoted 1 line
Is there any reason to use the BSP driver and not the one provided by Linux directly?
I was thinking the BSP driver is a *little bit* more updated than Linux one. Sometime it was a big difference between mainline and BSP driver. But now
the difference is not big and mostly in
DDR_BACKUPand WHITELIST support. I looked at mainline driver as well when implementing Xen driver.
What is the review process for patches to be merged in the BSP? Is it the same as Linux upstream?
Quoted 46 lines
Quoted 13 lines
[...]
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 106
Quoted 5 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0+#define dev_archdata (dev) ((struct ipmmu_vmsa_xen_device *)dev->archdata.iommu) + +/* Root/Cache IPMMU device's information */ +struct ipmmu_vmsa_device +{AFAICT, this file was converted to Xen coding style. If so, the style for struct is
Quoted 3 lines
struct ... {
...
};Yes, will correct everywhere in this file.
Quoted 17 lines
Quoted 17 lines
+ struct device *dev;
+ void__iomem*base;
+ structipmmu_vmsa_device*root;
+ structlist_headlist;
+ unsigned intnum_utlbs;
+ unsigned intnum_ctx;
+spinlock_tlock; /* Protects ctx and domains[] */ +DECLARE_BITMAP(ctx, IPMMU_CTX_MAX);
+ structipmmu_vmsa_domain*domains[IPMMU_CTX_MAX];xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7
-0+}; + +/* +* Root/Cache IPMMU domain's information +* +* Root IPMMU device is assigned to Root IPMMU domain while Cache IPMMU device +* is assigned to Cache IPMMU domain. Master devices are connected to Cache Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c
Quoted 6 lines
+ * IPMMU devices through specific ports called micro-TLBs.
+ * All Cache IPMMU devices, in turn, are connected to Root IPMMU device
+ * which manages IPMMU context.
+ */xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +2
-0+struct ipmmu_vmsa_domain +{Ditto.
Quoted 28 lines
ok
Quoted 25 lines
Quoted 13 lines
+ /*
+ * IPMMU device assigned to this IPMMU domain.
+ * Either Root device which is located at the main memory bus domain or + * Cache device which is located at each hierarchy bus domain. + */
+ structipmmu_vmsa_device*mmu;xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+ + /* Context used for this IPMMU domain */ + unsigned int context_id; + + /* Xen domain associated with this IPMMU domain */ + struct domain *d;xen/drivers/passthrough/arm/ipmmu-vmsa.c line 150
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ + /* The fields below are used for Cache IPMMU domain only */ + + /* + * Used to keep track of the master devices which are attached to this + * IPMMU domain (domain users). Master devices behind the same IPMMU device + * are grouped together by putting into the same IPMMU domain. + * Only when the refcount reaches 0 this IPMMU domain can be destroyed. + */ + int refcount;If the refcount cannot be 0, then I would prefer an unsigned int here.
Hrm, I meant cannot be negative.
Quoted 2 lines
It can be >= 0.
Ok, so please switch to unsigned int here please.
Quoted 32 lines
Quoted 23 lines
Quoted 23 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +22
-0+{ + return readl(mmu->base + offset); +} + +static void ipmmu_write (struct ipmmu_vmsa_device *mmu,unsigned int offset, + u32 data) +{ + writel(data, mmu->base + offset); +} + +static u32 ipmmu_ctx_read_root (struct ipmmu_vmsa_domain *domain, + unsigned int reg) +{ + return ipmmu_read(domain->mmu->root, + domain->context_id * IM_CTX_SIZE + reg); +} + +static void ipmmu_ctx_write_root (struct ipmmu_vmsa_domain *domain, + unsigned int reg, u32 data) +{ + ipmmu_write(domain->mmu->root, + domain->context_id * IM_CTX_SIZE + reg, data);Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 344
Quoted 6 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+} + +static void ipmmu_ctx_write_cache (struct ipmmu_vmsa_domain *domain, + unsigned int reg, u32 data) +{ + ASSERT(reg == IMCTR);What's the rationale of passing reg in parameter if it can only be equal to IMCTR?
Quoted 3 lines
Good question. I tried to retain the same interface as for
ipmmu_ctx_write_root(_all)for visibility.Cache IPMMU device has other than IMCTR context registers, but they are not used by this driver.
Could the function be able to deal with those other registers without any change?
Quoted 2 lines
Shall I drop reg parameter?
Let's discuss about it. See my question above.
Quoted 25 lines
Quoted 12 lines
Quoted 9 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+ + /* Mask fields which are implemented in IPMMU-MM only. */ + if ( !ipmmu_is_root(domain->mmu) ) + ipmmu_write(domain->mmu, domain->context_id * IM_CTX_SIZE + reg, + data & IMCTR_COMMON_MASK); +}[...]
Quoted 12 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 407
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+/* Enable MMU translation for the micro-TLB. */ +static void ipmmu_utlb_enable (struct ipmmu_vmsa_domain *domain, + unsigned int utlb) +{ + struct ipmmu_vmsa_device *mmu = domain->mmu; + + /* +* TODO: Reference-count the micro-TLB as several bus masters can be +* connected to the same micro-TLB. +*/ What would be the exact problem if this is not handled? Could a utlb shared between multiple domain?
Quoted 5 lines
Actually, this TODO as well as the whole "TLB and micro-TLB Management" code I tried to retain as much as possible.
The large amount of devices have unique micro-TLB connection, but there is a case, when not. At the moment, I don't see any problem if these potential devices (which share the same utlb) are assigned to the same Xen domain.
In this case we would just enable the same utlb more than once at the domain creation time (assign a device) and disable the same utlb more than once at the domain destruction time (deassign a device).
That's correct as long as we don't to support device hotplug. Platform device hotplug is not going to happen, this may happen for PCI devices.
Quoted 2 lines
The worst case scenario would be when these devices are assigned to different Xen domains. So, I think, the same utlb *can't* be shared between multiple Xen domains, since it points to the context bank to use for the page walk.
Thank you for the explanation. What can actually happen? Could it lead to a security issue (e.g the IPMMU is bypassed)?
Also, the question is whether this is worth to try to implement it. Do we have cases where devices use the same micro-TLB but assigned to different domains?
If not, then maybe you could just add check in the driver to prevent that use cases. The work around the iommu_group done by Paul [1] might be useful.
Anyway, from upstream perspective this is not a massive concern for now as platform device-passthrough is not security supported. So I would be happy if the TODO is addressed in a follow-up series.
[...]
Quoted 38 lines
Quoted 29 lines
Quoted 5 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0+/* Master devices management */ +static int ipmmu_attach_device (struct ipmmu_vmsa_domain *domain, + struct device *dev) +{ + struct ipmmu_vmsa_master_cfg *cfg = dev_archdata(dev)->cfg;xen/drivers/passthrough/arm/ipmmu-vmsa.c line 610
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ struct ipmmu_vmsa_device *mmu = cfg->mmu; + unsigned int i; + + if ( !mmu ) + { + dev_err(dev, "Cannot attach to IPMMU\n"); + return -ENXIO; + } + + if ( !domain->mmu )So you read
domain->mmuhere and ...Quoted 11 lines
+ {
+ /* The domain hasn't been used yet, initialize it. */
+domain->mmu= mmu;
+
+ /*
+ * We have already enabled context for Root IPMMU assigned to this + * Xen domain inipmmu_domain_init_context().
+ * Enable the context for Cache IPMMU only. Flush the TLB as required + * when modifying the context registers.
+ */
+ ipmmu_ctx_write_cache(domain, IMCTR,Quoted 8 lines
Quoted 6 lines
+
ipmmu_ctx_read_root(domain, IMCTR)|IMCTR_FLUSH);
+
+dev_info(dev, "Using IPMMU context %u\n", domain->context_id); + }
+ else if (domain->mmu!= mmu )... here. What actually promise that
domain->mmucan't change in parallel?
Quoted 1 line
ipmmu_attach_deviceis protected byxen_domain->lock
I find confusing to rely on xen_domain->lock to serialize access to a field from a different structure. It would be good if this is written in the document of the structure.
Also is this always read behind the same lock?
[...]
Quoted 9 lines
Quoted 4 lines
The IOMMU interface in Xen has not been designed with the new IOMMU bindings in mind. I would prefer if we look for extending
add_devicecallback to support platform device.This would allow to probe the device later on and therefore avoid to go through the device-tree multiple.
I completely agree with you that current implementation is not optimal and should be reworked in order not to scan the whole DT many times, but I am not completely understand what we should do and how exactly.
Could you, please, add more details?
It would be good to have an abstract way to add new device to IOMMU based on the generic IOMMU DT binding. I am quite keen to seen something similar to iommu_fwspec in Xen so this can be used for both DT and ACPI.
From an high level perspective, we would have some code add a new device to the IOMMU. The generic code would:
1) Parse the binding and prepare iommu_fwspec with the correct information
2) Call the IOMMU driver to register the new device
The new function would be either called from handle_device or a new loop over the DT nodes.
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +9
-0+ +static void ipmmu_device_reset (struct ipmmu_vmsa_device *mmu) +{ + unsigned int i; + + /* Disable all contexts. */ + for ( i = 0; i < mmu->num_ctx; ++i ) + ipmmu_write(mmu, i * IM_CTX_SIZE + IMCTR, 0); +}Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 784
Quoted 6 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+ +/* +* This function relies on the fact that Root IPMMU device is being probed +* the first. If not the case, it denies further Cache IPMMU device probes +* (returns the -ENODEV) until the Root IPMMU device has been registered +* for sure. Can we look at handling
-EDEFERin Xen instead?
Quoted 3 lines
I am not sure this is something we should implement at this stage (while only IPMMU driver would be a user). I have already resolved that possible issue by trying to locate a Root IPMMU device and probe it the first
to avoid the case described above. So now, we don't depend on how IPMMU devices are located in DT. Please, see
ipmmu_init(). So, I tend to live with it some time.
The reason I asked the question is the current solution feels like papering over an API that does not fit for the new driver. So it would be worth investigating whether a -EDEFER like could be easily used in Xen.
[...]
Quoted 18 lines
----------
Julien, what we should do with the fact that IPMMU supports only 3-level page table?
I left a TODO regarding that, but we need to work out some usable solution if possible.
/*
* As 4-level translation table is not supported in IPMMU, we need
* to check IPA size used for P2M table beforehand to be sure it is
* 3-level and the IPMMU will be able to use it.
*
* In case of using 4KB page granule we should use two concatenated
* translation tables at level 1 in order to support 40 bit IPA
* with 3-level translation table.
*
* TODO: Probably, when determing the "pa_range" insetup_virt_paging()
* we should take into the account the IPMMU ability as well.
Quoted 8 lines
*/
if (IPMMU_MAX_P2M_IPA_BITS<p2m_ipa_bits)
{
dev_err(&node->dev, "P2M IPA size is not supported (P2M=%u
IPMMU=%u)!\n",
p2m_ipa_bits,IPMMU_MAX_P2M_IPA_BITS); return-EOPNOTSUPP;
}
We have similar problem with the SMMU. The only strict requirement for the IOMMU is to have a valid p2m_ipa_bits at the time a domain is built. Note that the SMMU will store the value when probing the SMMU, but that could be reworked.
So rather than initializing the P2M first and then the IOMMU, I would first initialize the IOMMU so we can gather the requirements and then initialize the P2M.
In the P2M code, you can take into account the IOMMU requirements and further restrict if necessary. What do you think?
Cheers,
[1] <[email protected]>
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi, Julien.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Oleksandr Tyshchenko
Quoted 5 lines
From: Oleksandr Tyshchenko <[email protected]>
The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Do you have a link to the specification?
All I have is a TRM. Unfortunately, I can't share it.
Does anyone in the community has access to the spec?
Yes. I believe, there are persons from the Linux community who have access to the spec. I have already asked for review.
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
What are the major differences compare the Linux driver?
Well, the major differences are:
1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the page table with the CPU. Stage 1 translation is always bypassed in Xen driver.
Quoted 7 lines
Quoted 5 lines
So, Xen driver is supposed to be used with newest Gen3 SoC revisions only (H3 ES3.0, M3 ES3.0, etc.) which IPMMU hardware does support stage 2 translation table format.
2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.
3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.
Can this be written in the commit message? This is helpful for anyone reviewing the driver today and future developer.
Sure. Will update.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
+ * you can found at:
+ * url:git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git
+ * branch: v4.14.75-ltsi/rcar-3.9.2
+ * commit:a5266d298124874c2c06b8b13d073f6ecc2ee355
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Is there any reason to use the BSP driver and not the one provided by Linux directly?
I was thinking the BSP driver is a *little bit* more updated than Linux one. Sometime it was a big difference between mainline and BSP driver. But now
the difference is not big and mostly in
DDR_BACKUPand WHITELIST support. I looked at mainline driver as well when implementing Xen driver.What is the review process for patches to be merged in the BSP? Is it the same as Linux upstream?
I don't know at the moment, I will try to clarify this question.
Quoted 4 lines
Quoted 2 lines
It can be >= 0.
Ok, so please switch to unsigned int here please.
ok
Quoted 33 lines
Quoted 33 lines
Quoted 32 lines
Quoted 20 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0+{ + return readl(mmu->base + offset); +} + +static void ipmmu_write(struct ipmmu_vmsa_device *mmu, unsignedint offset,
+ u32 data)xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +12
-0+{ + writel(data, mmu->base + offset); +} + +static u32 ipmmu_ctx_read_root (struct ipmmu_vmsa_domain *domain, + unsigned int reg) +{ + return ipmmu_read(domain->mmu->root, + domain->context_id * IM_CTX_SIZE + reg); +} + +static void ipmmu_ctx_write_root(struct ipmmu_vmsa_domain *domain,xen/drivers/passthrough/arm/ipmmu-vmsa.c line 344
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ unsigned int reg, u32 data) +{ + ipmmu_write(domain->mmu->root, + domain->context_id * IM_CTX_SIZE + reg, data); +} + +static void ipmmu_ctx_write_cache (struct ipmmu_vmsa_domain *domain, + unsigned int reg, u32 data) +{ + ASSERT(reg == IMCTR);What's the rationale of passing reg in parameter if it can only be equal to IMCTR?
Quoted 5 lines
Quoted 3 lines
Good question. I tried to retain the same interface as for
ipmmu_ctx_write_root(_all)for visibility.Cache IPMMU device has other than IMCTR context registers, but they are not used by this driver.
Could the function be able to deal with those other registers without any change?
No. "data & IMCTR_COMMON_MASK" should be moved out of the function at least.
Quoted 4 lines
Quoted 2 lines
The worst case scenario would be when these devices are assigned to different Xen domains. So, I think, the same utlb *can't* be shared between multiple Xen domains, since it points to the context bank to use for the page walk.
Thank you for the explanation. What can actually happen? Could it lead to a security issue (e.g the IPMMU is bypassed)?
Yes, using context bank for IPA to PA translation from Domain A for a device running in Domain B could lead to something very bad. The best case scenario would be when IPA supplied by a device in Domain B is not effectively mapped in a context for Domain A. In this case we would just get a page fault...
Quoted 1 line
Also, the question is whether this is worth to try to implement it. Do we have cases where devices use the same micro-TLB but assigned to different domains?
No. At least I am not aware of. What is more that devices (which share the same utlb) can't be easily separated from each other the first in order to be assigned to different Xen domains then, I think.
Such DMA devices as AVB, SATA, eMMC/SD, USB, DU, GPU, etc (which really could be located in different Xen domains according to the particular use case) don't share utlbs.
Quoted 4 lines
If not, then maybe you could just add check in the driver to prevent that use cases. The work around the
iommu_groupdone by Paul [1] might be useful.Anyway, from upstream perspective this is not a massive concern for now as platform device-passthrough is not security supported. So I would be happy if the TODO is addressed in a follow-up series.
Agree.
So, the following actions:
1. TODO remains for this driver series.
2. TODO will be addressed in a follow-up series by *preventing* the use cases where the same utlb could be shared between multiple Xen domains.
Quoted 44 lines
[...]
Quoted 41 lines
Quoted 40 lines
Quoted 5 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0+/* Master devices management */ +static int ipmmu_attach_device (struct ipmmu_vmsa_domain *domain, + struct device *dev) +{ + struct ipmmu_vmsa_master_cfg *cfg = dev_archdata(dev)->cfg;xen/drivers/passthrough/arm/ipmmu-vmsa.c line 610
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ struct ipmmu_vmsa_device *mmu = cfg->mmu; + unsigned int i; + + if ( !mmu ) + { + dev_err(dev, "Cannot attach to IPMMU\n"); + return -ENXIO; + } + + if ( !domain->mmu )So you read
domain->mmuhere and ...Quoted 12 lines
+ {
+ /* The domain hasn't been used yet, initialize it. */ +domain->mmu= mmu;
+
+ /*
+ * We have already enabled context for Root IPMMU assigned to this
+ * Xen domain inipmmu_domain_init_context().
+ * Enable the context for Cache IPMMU only. Flush the TLB as required
+ * when modifying the context registers.
+ */Quoted 8 lines
+ ipmmu_ctx_write_cache(domain, IMCTR,
+ipmmu_ctx_read_root(domain, IMCTR)|IMCTR_FLUSH);
+
+dev_info(dev, "Using IPMMU context %u\n", domain->context_id);
+ }
+ else if (domain->mmu!= mmu )... here. What actually promise that
domain->mmucan't change in parallel?
Quoted 5 lines
Quoted 1 line
ipmmu_attach_deviceis protected byxen_domain->lockI find confusing to rely on
xen_domain->lockto serialize access to a field from a different structure. It would be good if this is written in the document of the structure.Also is this always read behind the same lock?
I think, yes. I couldn't find the case when not.
Will clarify documentation.
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
The IOMMU interface in Xen has not been designed with the new IOMMU bindings in mind. I would prefer if we look for extending
add_devicecallback to support platform device.This would allow to probe the device later on and therefore avoid to go through the device-tree multiple.
I completely agree with you that current implementation is not optimal and should be reworked in order not to scan the whole DT many times, but I am not completely understand what we should do and how exactly.
Could you, please, add more details?
Quoted 7 lines
It would be good to have an abstract way to add new device to IOMMU based on the generic IOMMU DT binding. I am quite keen to seen something similar to
iommu_fwspecin Xen so this can be used for both DT and ACPI.From an high level perspective, we would have some code add a new device to the IOMMU. The generic code would:
1) Parse the binding and prepareiommu_fwspecwith the correct information
2) Call the IOMMU driver to register the new deviceThe new function would be either called from
handle_deviceor a new loop over the DT nodes.
In the whole, I understand your point. I will come up with questions if any.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Can we look at handling
-EDEFERin Xen instead?I am not sure this is something we should implement at this stage (while only IPMMU driver would be a user). I have already resolved that possible issue by trying to locate a Root IPMMU device and probe it the first
to avoid the case described above. So now, we don't depend on how IPMMU devices are located in DT. Please, see
ipmmu_init(). So, I tend to live with it some time.The reason I asked the question is the current solution feels like papering over an API that does not fit for the new driver. So it would be worth investigating whether a
-EDEFERlike could be easily used in Xen.
Well, I will definitely investigate the possibility.
Quoted 26 lines
Quoted 17 lines
----------
Julien, what we should do with the fact that IPMMU supports only 3-level page table?
I left a TODO regarding that, but we need to work out some usable solution if possible.
/*
* As 4-level translation table is not supported in IPMMU, we need
* to check IPA size used for P2M table beforehand to be sure it is
* 3-level and the IPMMU will be able to use it.
*
* In case of using 4KB page granule we should use two concatenated
* translation tables at level 1 in order to support 40 bit IPA
* with 3-level translation table.
*
* TODO: Probably, when determing the "pa_range" insetup_virt_paging()Quoted 8 lines
* we should take into the account the IPMMU ability as well. */
if (IPMMU_MAX_P2M_IPA_BITS<p2m_ipa_bits)
{
dev_err(&node->dev, "P2M IPA size is not supported (P2M=%u IPMMU=%u)!\n",p2m_ipa_bits,IPMMU_MAX_P2M_IPA_BITS);
return-EOPNOTSUPP;
}
Quoted 5 lines
We have similar problem with the SMMU. The only strict requirement for the IOMMU is to have a valid
p2m_ipa_bitsat the time a domain is built. Note that the SMMU will store the value when probing the SMMU, but that could be reworked.So rather than initializing the P2M first and then the IOMMU, I would first initialize the IOMMU so we can gather the requirements and then initialize the P2M.
In the P2M code, you can take into account the IOMMU requirements and further restrict if necessary. What do you think?
I think, this sounds reasonable and worth trying. Could this TODO be addressed in a follow-up series?
Quoted 6 lines
Cheers,
[1] <[email protected]>
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Hi Oleksandr,
Quoted 35 lines · Oleksandr
Quoted 34 lines · Julien Grall
Quoted 34 lines
Quoted 33 lines · Oleksandr Tyshchenko
Quoted 20 lines
From: Oleksandr Tyshchenko <[email protected]>
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0+{ + return readl(mmu->base + offset); +} + +static void ipmmu_write(struct ipmmu_vmsa_device *mmu, unsignedint offset,
+ u32 data)xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +12
-0+{ + writel(data, mmu->base + offset); +} + +static u32 ipmmu_ctx_read_root (struct ipmmu_vmsa_domain *domain, + unsigned int reg) +{ + return ipmmu_read(domain->mmu->root, + domain->context_id * IM_CTX_SIZE + reg); +} + +static void ipmmu_ctx_write_root(struct ipmmu_vmsa_domain *domain,xen/drivers/passthrough/arm/ipmmu-vmsa.c line 344
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ unsigned int reg, u32 data) +{ + ipmmu_write(domain->mmu->root, + domain->context_id * IM_CTX_SIZE + reg, data); +} + +static void ipmmu_ctx_write_cache (struct ipmmu_vmsa_domain *domain, + unsigned int reg, u32 data) +{ + ASSERT(reg == IMCTR);What's the rationale of passing reg in parameter if it can only be equal to IMCTR?
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Good question. I tried to retain the same interface as for
ipmmu_ctx_write_root(_all)for visibility.Cache IPMMU device has other than IMCTR context registers, but they are not used by this driver.
Could the function be able to deal with those other registers without any change?
No. "data &
IMCTR_COMMON_MASK" should be moved out of the function at least.
I will leave up to you whether you want to drop the parameter. However, I'd like a comment explaining why the ASSERT() if you decide to keep it.
[...]
Quoted 6 lines
Quoted 4 lines
If not, then maybe you could just add check in the driver to prevent that use cases. The work around the
iommu_groupdone by Paul [1] might be useful.Anyway, from upstream perspective this is not a massive concern for now as platform device-passthrough is not security supported. So I would be happy if the TODO is addressed in a follow-up series.
Quoted 7 lines
Agree.
So, the following actions:
1. TODO remains for this driver series.
2. TODO will be addressed in a follow-up series by *preventing* the use cases where the same utlb could be shared between multiple Xen domains.
The 2 actions looks good to me.
[...]
Quoted 1 line
I think, this sounds reasonable and worth trying. Could this TODO be addressed in a follow-up series?
I am fine with that.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel