Add SMMUv3 Stage 1 Support for XEN guests
cover Add SMMUv3 Stage 1 Support for XEN guests
The SMMUv3 supports two stages of translation. Each stage of translation can be independently enabled. An incoming address is logically translated from VA to IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to the output PA.
Stage 1 is intended to be used by a software entity to provide isolation or translation to buffers within the entity, for example DMA isolation within an OS. Stage 2 is intended to be available in systems supporting the Virtualization Extensions and is intended to virtualize device DMA to guest VM address spaces. When both stage 1 and stage 2 are enabled, the translation configuration is called nested.
Stage 1 translation support is required to provide isolation between different devices within OS. XEN already supports Stage 2 translation but there is no support for Stage 1 translation. The goal of this work is to support Stage 1 translation for XEN guests. Stage 1 has to be configured within the guest to provide isolation.
We cannot trust the guest OS to control the SMMUv3 hardware directly as compromised guest OS can corrupt the SMMUv3 configuration and make the system vulnerable. The guest gets the ownership of the stage 1 page tables and also owns stage 1 configuration structures. The XEN handles the root configuration structure (for security reasons), including the stage 2 configuration.
XEN will emulate the SMMUv3 hardware and exposes the virtual SMMUv3 to the guest. Guest can use the native SMMUv3 driver to configure the stage 1 translation. When the guest configures the SMMUv3 for Stage 1, XEN will trap the access and configure hardware.
SMMUv3 Driver(Guest OS) -> Configure the Stage-1 translation -> XEN trap access -> XEN SMMUv3 driver configure the HW.
SMMUv3 driver has to be updated to support the Stage-1 translation support based on work done by the KVM team to support Nested Stage translation:
https://github.com/eauger/linux/commits/v5.11-stallv12-2stage-v14
https://lwn.net/Articles/852299/
As the stage 1 translation is configured by XEN on behalf of the guest, translation faults encountered during the translation process need to be propagated up to the guest and re-injected into the guest. When the guest invalidates stage 1 related caches, invalidations must be forwarded to the SMMUv3 hardware.
This patch series is sent as RFC to get the initial feedback from the community. This patch series consists of 21 patches which is a big number for the reviewer to review the patches but to understand the feature end-to-end we thought of sending this as a big series. Once we will get initial feedback, we will divide the series into a small number of patches for review.
Jean-Philippe Brucker (1):
xen/arm: smmuv3: Maintain a SID->device structure
Rahul Singh (20):
xen/arm: smmuv3: Add support for stage-1 and nested stage translation xen/arm: smmuv3: Alloc io_domain for each device
xen/arm: vIOMMU: add generic vIOMMU framework
xen/arm: vsmmuv3: Add dummy support for virtual SMMUv3 for guests xen/domctl: Add XEN_DOMCTL_CONFIG_VIOMMU_* and viommu config param xen/arm: vIOMMU: Add cmdline boot option "viommu = <boolean>" xen/arm: vsmmuv3: Add support for registers emulation xen/arm: vsmmuv3: Add support for cmdqueue handling xen/arm: vsmmuv3: Add support for command CMD_CFGI_STE xen/arm: vsmmuv3: Attach Stage-1 configuration to SMMUv3 hardware xen/arm: vsmmuv3: Add support for event queue and global error xen/arm: vsmmuv3: Add "iommus" property node for dom0 devices xen/arm: vIOMMU: IOMMU device tree node for dom0
xen/arm: vsmmuv3: Emulated SMMUv3 device tree node for dom0less arm/libxl: vsmmuv3: Emulated SMMUv3 device tree node in libxl xen/arm: vsmmuv3: Alloc virq for virtual SMMUv3
xen/arm: iommu: skip the iommu-map property for PCI devices xen/arm: vsmmuv3: Add support to send stage-1 event to guest libxl/arm: vIOMMU: Modify the partial device tree for iommus xen/arm: vIOMMU: Modify the partial device tree for dom0less
docs/man/xl.cfg.5.pod.in | 11 + docs/misc/xen-command-line.pandoc | 7 + tools/golang/xenlight/helpers.gen.go | 2 + tools/golang/xenlight/types.gen.go | 1 + tools/include/libxl.h | 5 + tools/libs/light/libxl_arm.c | 121 +++- tools/libs/light/libxl_types.idl | 6 + tools/xl/xl_parse.c | 9 + xen/arch/arm/domain.c | 18 + xen/arch/arm/domain_build.c | 213 +++++- xen/arch/arm/include/asm/domain.h | 4 + xen/arch/arm/include/asm/viommu.h | 102 +++ xen/drivers/passthrough/Kconfig | 14 + xen/drivers/passthrough/arm/Makefile | 2 + xen/drivers/passthrough/arm/smmu-v3.c | 370 +++++++++-- xen/drivers/passthrough/arm/smmu-v3.h | 43 +- xen/drivers/passthrough/arm/viommu.c | 87 +++ xen/drivers/passthrough/arm/vsmmu-v3.c | 887 +++++++++++++++++++++++++ xen/drivers/passthrough/arm/vsmmu-v3.h | 32 + xen/include/public/arch-arm.h | 14 +- xen/include/public/device_tree_defs.h | 1 + xen/include/xen/iommu.h | 14 + 22 files changed, 1913 insertions(+), 50 deletions(-) create mode 100644 xen/arch/arm/include/asm/viommu.h create mode 100644 xen/drivers/passthrough/arm/viommu.c create mode 100644 xen/drivers/passthrough/arm/vsmmu-v3.c create mode 100644 xen/drivers/passthrough/arm/vsmmu-v3.h
--
2.25.1
RFC PATCH 1/21 xen/arm: smmuv3: Maintain a SID->device structure
From: Jean-Philippe Brucker <[email protected]>
Backport Linux commit cdf315f907d4. This is the clean backport without any changes.
When handling faults from the event or PRI queue, we need to find the struct device associated with a SID. Add a rb_tree to keep track of SIDs.
Origin: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git cdf315f907d4
--- xen/drivers/passthrough/arm/smmu-v3.c | 131 +++++++++++++++++++++----- xen/drivers/passthrough/arm/smmu-v3.h | 13 ++- 2 files changed, 118 insertions(+), 26 deletions(-)
xen/drivers/passthrough/arm/smmu-v3.c +107 -24
@@ -810,6 +810,27 @@ static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid)return 0 ; } +__maybe_unused +static struct arm_smmu_master * +arm_smmu_find_master (struct arm_smmu_device *smmu,u32 sid) +{ +struct rb_node *node; +struct arm_smmu_stream *stream; + + node = smmu->streams.rb_node; +while (node) { + stream = rb_entry(node, struct arm_smmu_stream, node); +if (stream->id < sid) + node = node->rb_right; +else if (stream->id > sid) + node = node->rb_left; +else +return stream->master; + } + +return NULL ; +} +/* IRQ and event handlers */ static void arm_smmu_evtq_tasklet (void *dev) {
@@ -1047,8 +1068,8 @@ static int arm_smmu_atc_inv_master(struct arm_smmu_master *master,if (!master->ats_enabled)return 0 ; -for (i =0 ; i < master->num_sids; i++) { - cmd->atc.sid = master->sids[i]; +for (i =0 ; i < master->num_streams; i++) { + cmd->atc.sid = master->streams[i].id;arm_smmu_cmdq_issue_cmd (master->smmu, cmd); }
@@ -1276,13 +1297,13 @@ static void arm_smmu_install_ste_for_dev(struct arm_smmu_master *master)int i, j;struct arm_smmu_device *smmu = master->smmu; -for (i =0 ; i < master->num_sids; ++i) { -u32 sid = master->sids[i]; +for (i =0 ; i < master->num_streams; ++i) { +u32 sid = master->streams[i].id;__le64 *step =arm_smmu_get_step_for_sid (smmu, sid);/* Bridged PCI devices may end up with duplicated IDs */ for (j =0 ; j < i; j++) -if (master->sids[j] == sid) +if (master->streams[j].id == sid)break ;if (j < i)continue ;
@@ -1489,12 +1510,86 @@ static bool arm_smmu_sid_in_range(struct arm_smmu_device *smmu, u32 sid)return sid < limit; } + +static int arm_smmu_insert_master (struct arm_smmu_device *smmu, +struct arm_smmu_master *master) +{ +int i; +int ret =0 ; +struct arm_smmu_stream *new_stream, *cur_stream; +struct rb_node **new_node, *parent_node =NULL ; +struct iommu_fwspec *fwspec =dev_iommu_fwspec_get (master->dev); + + master->streams =_xzalloc_array (sizeof (*master->streams),sizeof (void *), + fwspec->num_ids); +if (!master->streams) +return -ENOMEM ; + master->num_streams = fwspec->num_ids; + +mutex_lock (&smmu->streams_mutex); +for (i =0 ; i < fwspec->num_ids; i++) { +u32 sid = fwspec->ids[i]; + + new_stream = &master->streams[i]; + new_stream->id = sid; + new_stream->master = master; + +/* +* Check the SIDs are in range of the SMMU and our stream table +*/ +if (!arm_smmu_sid_in_range (smmu, sid)) { + ret = -ERANGE ; +break ; + } + +/* Ensure l2 strtab is initialised */ +if (smmu->features &ARM_SMMU_FEAT_2_LVL_STRTAB ) { + ret =arm_smmu_init_l2_strtab (smmu, sid); +if (ret) +break ; + } + +/* Insert into SID tree */ + new_node = &(smmu->streams.rb_node); +while (*new_node) { + cur_stream = rb_entry(*new_node, struct arm_smmu_stream, + node); + parent_node = *new_node; +if (cur_stream->id > new_stream->id) { + new_node = &((*new_node)->rb_left); + }else if (cur_stream->id < new_stream->id) { + new_node = &((*new_node)->rb_right); + }else { +dev_warn (master->dev, +"stream %u already in tree\n" , + cur_stream->id); + ret = -EINVAL ; +break ; + } + } +if (ret) +break ; + +rb_link_node (&new_stream->node, parent_node, new_node); +rb_insert_color (&new_stream->node, &smmu->streams); + } + +if (ret) { +for (i--; i >=0 ; i--) +rb_erase (&master->streams[i].node, &smmu->streams); +xfree (master->streams); + } +mutex_unlock (&smmu->streams_mutex); + +return ret; +} +/* Forward declaration */ static struct arm_smmu_device *arm_smmu_get_by_dev (struct device *dev);static int arm_smmu_add_device (u8 devfn,struct device *dev) { -int i, ret; +int ret;struct arm_smmu_device *smmu;struct arm_smmu_master *master;struct iommu_fwspec *fwspec =dev_iommu_fwspec_get (dev);
@@ -1512,26 +1607,11 @@ static int arm_smmu_add_device(u8 devfn, struct device *dev) master->dev = dev; master->smmu = smmu; - master->sids = fwspec->ids; - master->num_sids = fwspec->num_ids;dev_iommu_priv_set (dev, master); -/* Check the SIDs are in range of the SMMU and our stream table */ -for (i =0 ; i < master->num_sids; i++) { -u32 sid = master->sids[i]; - -if (!arm_smmu_sid_in_range (smmu, sid)) { - ret = -ERANGE ; - goto err_free_master; - } - -/* Ensure l2 strtab is initialised */ -if (smmu->features &ARM_SMMU_FEAT_2_LVL_STRTAB ) { - ret =arm_smmu_init_l2_strtab (smmu, sid); -if (ret) - goto err_free_master; - } - } + ret =arm_smmu_insert_master (smmu, master); +if (ret) + goto err_free_master;/* * Note that PASID must be enabled before, and disabled after ATS:
@@ -1752,6 +1832,9 @@ static int arm_smmu_init_structures(struct arm_smmu_device *smmu) {int ret; +mutex_init (&smmu->streams_mutex); + smmu->streams =RB_ROOT ; + ret =arm_smmu_init_queues (smmu);if (ret)return ret;
xen/drivers/passthrough/arm/smmu-v3.h +11 -2
@@ -637,6 +637,15 @@ struct arm_smmu_device {struct tasklet evtq_irq_tasklet;struct tasklet priq_irq_tasklet;struct tasklet combined_irq_tasklet; + +struct rb_root streams; +struct mutex streams_mutex; +}; + +struct arm_smmu_stream { +u32 id; +struct arm_smmu_master *master; +struct rb_node node; };/* SMMU private data for each master */
@@ -645,8 +654,8 @@ struct arm_smmu_master {struct device *dev;struct arm_smmu_domain *domain;struct list_head domain_head; -u32 *sids; -unsigned int num_sids; +struct arm_smmu_stream *streams; +unsigned int num_streams;bool ats_enabled; };
Signature · 1 line
2.25.1
RFC PATCH 2/21 xen/arm: smmuv3: Add support for stage-1 and nested stage translation
In reply to Rahul SinghXen SMMUv3 driver only supports stage-2 translation. Add support for Stage-1 translation that is required to support nested stage translation.
In true nested mode, both s1_cfg and s2_cfg will coexist. Let's remove the union. When nested stage translation is setup, both s1_cfg and s2_cfg are valid.
We introduce a new smmu_domain abort field that will be set upon guest stage-1 configuration passing. If no guest stage-1 config has been attached, it is ignored when writing the STE.
arm_smmu_write_strtab_ent() is modified to write both stage fields in the STE and deal with the abort field.
--- xen/drivers/passthrough/arm/smmu-v3.c | 94 +++++++++++++++++++++++---- xen/drivers/passthrough/arm/smmu-v3.h | 9 +++ 2 files changed, 92 insertions(+), 11 deletions(-)
xen/drivers/passthrough/arm/smmu-v3.c +83 -11
@@ -686,8 +686,10 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,* 3. Update Config, sync */ u64 val =le64_to_cpu (dst[0 ]); -bool ste_live = false; +bool s1_live = false, s2_live = false, ste_live = false; +bool abort, translate = false;struct arm_smmu_device *smmu =NULL ; +struct arm_smmu_s1_cfg *s1_cfg =NULL ;struct arm_smmu_s2_cfg *s2_cfg =NULL ;struct arm_smmu_domain *smmu_domain =NULL ;struct arm_smmu_cmdq_ent prefetch_cmd = {
@@ -702,30 +704,54 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, smmu = master->smmu; } -if (smmu_domain) - s2_cfg = &smmu_domain->s2_cfg; +if (smmu_domain) { +switch (smmu_domain->stage) { +case ARM_SMMU_DOMAIN_NESTED : + s1_cfg = &smmu_domain->s1_cfg; + fallthrough; +case ARM_SMMU_DOMAIN_S2 : + s2_cfg = &smmu_domain->s2_cfg; +break ; +default : +break ; + } + translate = !!s1_cfg || !!s2_cfg; + }if (val &STRTAB_STE_0_V ) {switch (FIELD_GET (STRTAB_STE_0_CFG , val)) {case STRTAB_STE_0_CFG_BYPASS :break ; +case STRTAB_STE_0_CFG_S1_TRANS : + s1_live = true; +break ;case STRTAB_STE_0_CFG_S2_TRANS : - ste_live = true; + s2_live = true; +break ; +case STRTAB_STE_0_CFG_NESTED : + s1_live = true; + s2_live = true;break ;case STRTAB_STE_0_CFG_ABORT : -BUG_ON (!disable_bypass);break ;default :BUG ();/* STE corruption */ } } + ste_live = s1_live || s2_live; +/* Nuke the existing STE_0 value, as we're going to rewrite it */ val =STRTAB_STE_0_V ;/* Bypass/fault */ -if (!smmu_domain || !(s2_cfg)) { -if (!smmu_domain && disable_bypass) +if (!smmu_domain) + abort = disable_bypass; +else + abort = smmu_domain->abort; + +if (abort || !translate) { +if (abort) val |=FIELD_PREP (STRTAB_STE_0_CFG ,STRTAB_STE_0_CFG_ABORT );else val |=FIELD_PREP (STRTAB_STE_0_CFG ,STRTAB_STE_0_CFG_BYPASS );
@@ -743,8 +769,39 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,return ; } +if (ste_live) { +/* First invalidate the live STE */ + dst[0 ] =cpu_to_le64 (STRTAB_STE_0_CFG_ABORT ); +arm_smmu_sync_ste_for_sid (smmu, sid); + } + +if (s1_cfg) { +BUG_ON (s1_live); + dst[1 ] =cpu_to_le64 ( +FIELD_PREP (STRTAB_STE_1_S1DSS ,STRTAB_STE_1_S1DSS_SSID0 ) | +FIELD_PREP (STRTAB_STE_1_S1CIR ,STRTAB_STE_1_S1C_CACHE_WBRA ) | +FIELD_PREP (STRTAB_STE_1_S1COR ,STRTAB_STE_1_S1C_CACHE_WBRA ) | +FIELD_PREP (STRTAB_STE_1_S1CSH ,ARM_SMMU_SH_ISH ) | +FIELD_PREP (STRTAB_STE_1_STRW ,STRTAB_STE_1_STRW_NSEL1 )); + +if (smmu->features &ARM_SMMU_FEAT_STALLS && + !(smmu->features &ARM_SMMU_FEAT_STALL_FORCE )) + dst[1 ] |=cpu_to_le64 (STRTAB_STE_1_S1STALLD ); + + val |= (s1_cfg->s1ctxptr &STRTAB_STE_0_S1CTXPTR_MASK ) | +FIELD_PREP (STRTAB_STE_0_CFG ,STRTAB_STE_0_CFG_S1_TRANS ) | +FIELD_PREP (STRTAB_STE_0_S1CDMAX , s1_cfg->s1cdmax) | +FIELD_PREP (STRTAB_STE_0_S1FMT , s1_cfg->s1fmt); + } +if (s2_cfg) { -BUG_ON (ste_live); +u64 vttbr = s2_cfg->vttbr &STRTAB_STE_3_S2TTB_MASK ; + +if (s2_live) { +u64 s2ttb =le64_to_cpu (dst[3 ]) &STRTAB_STE_3_S2TTB_MASK ; +BUG_ON (s2ttb != vttbr); + } + dst[2] = cpu_to_le64( FIELD_PREP(STRTAB_STE_2_S2VMID, s2_cfg->vmid) | FIELD_PREP(STRTAB_STE_2_VTCR, s2_cfg->vtcr) |
@@ -754,9 +811,12 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,STRTAB_STE_2_S2PTW |STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2R); - dst[3 ] =cpu_to_le64 (s2_cfg->vttbr &STRTAB_STE_3_S2TTB_MASK ); + dst[3 ] =cpu_to_le64 (vttbr); val |=FIELD_PREP (STRTAB_STE_0_CFG ,STRTAB_STE_0_CFG_S2_TRANS ); + }else { + dst[2 ] =0 ; + dst[3 ] =0 ; }if (master->ats_enabled)
@@ -1259,6 +1319,15 @@ static int arm_smmu_domain_finalise(struct iommu_domain *domain, {int ret;struct arm_smmu_domain *smmu_domain =to_smmu_domain (domain); +struct arm_smmu_device *smmu = smmu_domain->smmu; + +if (smmu_domain->stage ==ARM_SMMU_DOMAIN_NESTED && + (!(smmu->features &ARM_SMMU_FEAT_TRANS_S1 ) || + !(smmu->features &ARM_SMMU_FEAT_TRANS_S2 ))) { +dev_info (smmu_domain->smmu->dev, +"does not implement two stages\n" ); +return -EINVAL ; + }/* Restrict the stage to what we can actually support */ smmu_domain->stage =ARM_SMMU_DOMAIN_S2 ;
@@ -2307,11 +2376,14 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu) smmu->features |=ARM_SMMU_FEAT_STALLS ; } +if (reg &IDR0_S1P ) + smmu->features |=ARM_SMMU_FEAT_TRANS_S1 ; +if (reg &IDR0_S2P ) smmu->features |=ARM_SMMU_FEAT_TRANS_S2 ; -if (!(reg &IDR0_S2P )) { -dev_err (smmu->dev,"no stage-2 translation support!\n" ); +if (!(reg & (IDR0_S1P |IDR0_S2P ))) { +dev_err (smmu->dev,"no translation support!\n" );return -ENXIO ; }
xen/drivers/passthrough/arm/smmu-v3.h +9 -0
@@ -197,6 +197,7 @@#define STRTAB_STE_0_CFG_BYPASS 4#define STRTAB_STE_0_CFG_S1_TRANS 5#define STRTAB_STE_0_CFG_S2_TRANS 6 +#define STRTAB_STE_0_CFG_NESTED 7#define STRTAB_STE_0_S1FMT GENMASK_ULL(5, 4)#define STRTAB_STE_0_S1FMT_LINEAR 0
@@ -547,6 +548,12 @@ struct arm_smmu_strtab_l1_desc {dma_addr_t l2ptr_dma; }; +struct arm_smmu_s1_cfg { +u64 s1ctxptr; +u8 s1fmt; +u8 s1cdmax; +}; +struct arm_smmu_s2_cfg {u16 vmid;u64 vttbr;
@@ -667,7 +674,9 @@ struct arm_smmu_domain {atomic_t nr_ats_masters;enum arm_smmu_domain_stage stage; +struct arm_smmu_s1_cfg s1_cfg;struct arm_smmu_s2_cfg s2_cfg; +bool abort;/* Xen domain associated with this SMMU domain */ struct domain *d;
Signature · 1 line
2.25.1
RFC PATCH 3/21 xen/arm: smmuv3: Alloc io_domain for each device
In reply to Rahul SinghIn current implementation io_domain is allocated once for each xen domain as Stage2 translation is common for all devices in same xen domain.
Nested stage supports S1 and S2 configuration at the same time. Stage1 translation will be different for each device as linux kernel will allocate page-table for each device.
Alloc io_domain for each device so that each device can have different Stage-1 and Stage-2 configuration structure.
--- xen/drivers/passthrough/arm/smmu-v3.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-)
xen/drivers/passthrough/arm/smmu-v3.c +11 -2
@@ -2753,11 +2753,13 @@ static struct arm_smmu_device *arm_smmu_get_by_dev(struct device *dev)static struct iommu_domain *arm_smmu_get_domain (struct domain *d,struct device *dev) { +unsigned long flags;struct iommu_domain *io_domain;struct arm_smmu_domain *smmu_domain;struct iommu_fwspec *fwspec =dev_iommu_fwspec_get (dev);struct arm_smmu_xen_domain *xen_domain =dom_iommu (d)->arch.priv;struct arm_smmu_device *smmu =arm_smmu_get_by_dev (fwspec->iommu_dev); +struct arm_smmu_master *master;if (!smmu)return NULL ;
@@ -2768,8 +2770,15 @@ static struct iommu_domain *arm_smmu_get_domain(struct domain *d,*/ list_for_each_entry (io_domain, &xen_domain->contexts, list) { smmu_domain =to_smmu_domain (io_domain); -if (smmu_domain->smmu == smmu) -return io_domain; + +spin_lock_irqsave (&smmu_domain->devices_lock, flags); +list_for_each_entry (master, &smmu_domain->devices, domain_head) { +if (master->dev == dev) { +spin_unlock_irqrestore (&smmu_domain->devices_lock, flags); +return io_domain; + } + } +spin_unlock_irqrestore (&smmu_domain->devices_lock, flags); }return NULL ; }
Signature · 1 line
2.25.1
RFC PATCH 4/21 xen/arm: vIOMMU: add generic vIOMMU framework
In reply to Rahul SinghThis patch adds basic framework for vIOMMU.
--- xen/arch/arm/domain.c | 17 +++++++ xen/arch/arm/domain_build.c | 3 ++ xen/arch/arm/include/asm/viommu.h | 70 ++++++++++++++++++++++++++++ xen/drivers/passthrough/Kconfig | 6 +++ xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/viommu.c | 48 +++++++++++++++++++ xen/include/public/arch-arm.h | 4 ++ 7 files changed, 149 insertions(+) create mode 100644 xen/arch/arm/include/asm/viommu.h create mode 100644 xen/drivers/passthrough/arm/viommu.c
xen/arch/arm/domain.c +17 -0
@@ -37,6 +37,7 @@#include <asm/tee/tee.h> #include <asm/vfp.h> #include <asm/vgic.h> +#include <asm/viommu.h> #include <asm/vtimer.h> #include "vpci.h"
@@ -691,6 +692,13 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +if ( config->arch.viommu_type !=XEN_DOMCTL_CONFIG_VIOMMU_NONE ) + { +dprintk (XENLOG_INFO , +"vIOMMU type requested not supported by the platform or Xen\n" ); +return -EINVAL ; + } +return 0 ; }
@@ -783,6 +791,9 @@ int arch_domain_create(struct domain *d,if ( (rc =domain_vpci_init (d)) !=0 ) goto fail; +if ( (rc =domain_viommu_init (d, config->arch.viommu_type)) !=0 ) + goto fail; +return 0 ; fail:
@@ -998,6 +1009,7 @@ static int relinquish_memory(struct domain *d, struct page_list_head *list)enum { PROG_pci =1 , PROG_tee, + PROG_viommu, PROG_xen, PROG_page, PROG_mapping,
@@ -1048,6 +1060,11 @@ int domain_relinquish_resources(struct domain *d)if (ret )return ret; + PROGRESS(viommu): + ret =viommu_relinquish_resources (d); +if (ret ) +return ret; + PROGRESS(xen): ret =relinquish_memory (d, &d->xenpage_list);if ( ret )
xen/arch/arm/domain_build.c +3 -0
@@ -27,6 +27,7 @@#include <asm/setup.h> #include <asm/cpufeature.h> #include <asm/domain_build.h> +#include <asm/viommu.h> #include <xen/event.h> #include <xen/irq.h>
@@ -3858,6 +3859,7 @@ void __init create_domUs(void)struct domain *d;struct xen_domctl_createdomain d_cfg = { .arch.gic_version =XEN_DOMCTL_CONFIG_GIC_NATIVE , + .arch.viommu_type =viommu_get_type (), .flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap,/* * The default of 1023 should be sufficient for guests because
@@ -4052,6 +4054,7 @@ void __init create_dom0(void)printk (XENLOG_WARNING "Maximum number of vGIC IRQs exceeded.\n" ); dom0_cfg.arch.tee_type =tee_get_type (); dom0_cfg.max_vcpus =dom0_max_vcpus (); + dom0_cfg.arch.viommu_type =viommu_get_type ();if ( iommu_enabled ) dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
xen/arch/arm/include/asm/viommu.h new file +70 -0
70 lines not on this page · raw message
xen/drivers/passthrough/Kconfig +6 -0
@@ -35,6 +35,12 @@ config IPMMU_VMSA (H3 ES3.0, M3-W+, etc) or Gen4 SoCs which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is. +config VIRTUAL_IOMMU + bool "Virtual IOMMU Support (UNSUPPORTED)" if UNSUPPORTED + default n + help + Support virtual IOMMU infrastructure to implement vIOMMU. + endif config IOMMU_FORCE_PT_SHARE
xen/drivers/passthrough/arm/Makefile +1 -0
@@ -2,3 +2,4 @@ obj-y += iommu.o iommu_helpers.o iommu_fwspec.o obj-$(CONFIG_ARM_SMMU) += smmu.o obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o obj-$(CONFIG_ARM_SMMU_V3) += smmu-v3.o +obj-$(CONFIG_VIRTUAL_IOMMU) += viommu.o
xen/drivers/passthrough/arm/viommu.c new file +48 -0
48 lines not on this page · raw message
xen/include/public/arch-arm.h +4 -0
@@ -297,10 +297,14 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);#define XEN_DOMCTL_CONFIG_TEE_NONE 0#define XEN_DOMCTL_CONFIG_TEE_OPTEE 1 +#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0 +struct xen_arch_domainconfig {/* IN/OUT */ uint8_t gic_version;/* IN */ +uint8_t viommu_type; +/* IN */ uint16_t tee_type;/* IN */ uint32_t nr_spis;
Signature · 1 line
2.25.1
Re: [RFC PATCH 04/21] xen/arm: vIOMMU: add generic vIOMMU framework
xen/drivers/passthrough/Kconfig config IPMMU_VMSA · line 42
Quoted 11 lines · Rahul Singh
xen/drivers/passthrough/Kconfig +5
-0@@ -35,6 +35,12 @@ config IPMMU_VMSA (H3 ES3.0, M3-W+, etc) or Gen4 SoCs which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is. +config VIRTUAL_IOMMU + bool "Virtual IOMMU Support (UNSUPPORTED)" if UNSUPPORTED + default n + help + Support virtual IOMMU infrastructure to implement vIOMMU.
I simply "virtual" specific enough in the name? Seeing that there are multiple IOMMU flavors for Arm, and judging from the titles of subsequent patches, you're implementing a virtualized form of only one variant.
Also, nit: Please omit "default n" here - it leads to a needless line in the resulting .config, which in addition prevents the prompt from appearing for user selection when someone later enables UNSUPPORTED in their config and then runs e.g. "make oldconfig". But perhaps you anyway really mean
config VIRTUAL_IOMMU
bool "Virtual IOMMU Support (UNSUPPORTED)" depends on UNSUPPORTED
help
Support virtual IOMMU infrastructure to implement vIOMMU.
?
Note (nit again) the slightly altered indentation I'm also using in the alternative suggestion.
Jan
Re: [RFC PATCH 04/21] xen/arm: vIOMMU: add generic vIOMMU framework
Hi Jan,
Quoted 9 lines · Jan Beulich
Quoted 6 lines · Rahul Singh
xen/drivers/passthrough/Kconfig +0
-0@@ -35,6 +35,12 @@ config IPMMU_VMSA (H3 ES3.0, M3-W+, etc) or Gen4 SoCs which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is.
Quoted 9 lines
xen/drivers/passthrough/Kconfig config IPMMU_VMSA · line 42
Quoted 5 lines
xen/drivers/passthrough/Kconfig +5
-0+config VIRTUAL_IOMMU + bool "Virtual IOMMU Support (UNSUPPORTED)" if UNSUPPORTED + default n + help + Support virtual IOMMU infrastructure to implement vIOMMU.I simply "virtual" specific enough in the name? Seeing that there are multiple IOMMU flavors for Arm, and judging from the titles of subsequent patches, you're implementing a virtualized form of only one variant.
I agree with you I will remove the virtual in next version.
Quoted 13 lines
Also, nit: Please omit "default n" here - it leads to a needless line in the resulting .config, which in addition prevents the prompt from appearing for user selection when someone later enables UNSUPPORTED in their config and then runs e.g. "make oldconfig". But perhaps you anyway really mean
config
VIRTUAL_IOMMU
bool "Virtual IOMMU Support (UNSUPPORTED)" depends on UNSUPPORTED
help
Support virtual IOMMU infrastructure to implement vIOMMU.
Quoted 5 lines
?
Note (nit again) the slightly altered indentation I'm also using in the alternative suggestion.
I will modify as below:
config VIRTUAL_IOMMU
bool "Virtual IOMMU Support (UNSUPPORTED)”
depends on UNSUPPORTED
help
Support IOMMU infrastructure to implement different variants of virtual IOMMUs.
Regards,
Rahul
Re: [RFC PATCH 04/21] xen/arm: vIOMMU: add generic vIOMMU framework
In reply to Rahul SinghHi Rahul,
Quoted 60 lines · Rahul Singh
This patch adds basic framework for vIOMMU.
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/domain.c | 17 +++++++ xen/arch/arm/domain_build.c | 3 ++ xen/arch/arm/include/asm/viommu.h | 70 ++++++++++++++++++++++++++++ xen/drivers/passthrough/Kconfig | 6 +++ xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/viommu.c | 48 +++++++++++++++++++ xen/include/public/arch-arm.h | 4 ++ 7 files changed, 149 insertions(+)create mode 100644
xen/arch/arm/include/asm/viommu.hcreate mode 100644xen/drivers/passthrough/arm/viommu.cxen/arch/arm/domain.c +12
-0@@ -37,6 +37,7 @@#include <asm/tee/tee.h> #include <asm/vfp.h> #include <asm/vgic.h> +#include <asm/viommu.h> #include <asm/vtimer.h> #include "vpci.h" @@ -691,6 +692,13 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +if ( config->arch.viommu_type !=XEN_DOMCTL_CONFIG_VIOMMU_NONE ) + { +dprintk (XENLOG_INFO , +"vIOMMU type requested not supported by the platform or Xen\n" ); +return -EINVAL ; + } +return 0 ; }@@ -783,6 +791,9 @@ int arch_domain_create(struct domain *d,if ( (rc =domain_vpci_init (d)) !=0 ) goto fail; +if ( (rc =domain_viommu_init (d, config->arch.viommu_type)) !=0 ) + goto fail; +return 0 ; fail:@@ -998,6 +1009,7 @@ static int relinquish_memory(struct domain *d, struct page_list_head *list)enum { PROG_pci =1 , PROG_tee, + PROG_viommu, PROG_xen, PROG_page, PROG_mapping,
xen/arch/arm/domain.c int domain_relinquish_resources(struct domain *d) · line 1066
Quoted 7 lines
xen/arch/arm/domain.c +4
-0@@ -1048,6 +1060,11 @@ int domain_relinquish_resources(struct domain *d)if (ret )return ret; + PROGRESS(viommu): + ret =viommu_relinquish_resources (d); +if (ret ) +return ret;
I would have expected us to relinquish the vIOMMU resources *before* we detach the devices. So can you explain the ordering?
Quoted 17 lines
xen/arch/arm/domain.c +1
-0+ PROGRESS(xen): ret =relinquish_memory (d, &d->xenpage_list);if ( ret )xen/arch/arm/domain_build.c +1
-0@@ -27,6 +27,7 @@#include <asm/setup.h> #include <asm/cpufeature.h> #include <asm/domain_build.h> +#include <asm/viommu.h> #include <xen/event.h> #include <xen/irq.h>
xen/arch/arm/domain_build.c void __init create_domUs(void) · line 3862
Quoted 4 lines
xen/arch/arm/domain_build.c +1
-0@@ -3858,6 +3859,7 @@ void __init create_domUs(void)struct domain *d;struct xen_domctl_createdomain d_cfg = { .arch.gic_version =XEN_DOMCTL_CONFIG_GIC_NATIVE , + .arch.viommu_type =viommu_get_type (),
I don't think the vIOMMU should be enabled to dom0less domU by default.
xen/arch/arm/domain_build.c void __init create_dom0(void) · line 4057
Quoted 8 lines
xen/arch/arm/domain_build.c +1
-0.flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap,/* * The default of 1023 should be sufficient for guests because @@ -4052,6 +4054,7 @@ void __init create_dom0(void)printk (XENLOG_WARNING "Maximum number of vGIC IRQs exceeded.\n" ); dom0_cfg.arch.tee_type =tee_get_type (); dom0_cfg.max_vcpus =dom0_max_vcpus (); + dom0_cfg.arch.viommu_type =viommu_get_type ();
Same here.
Quoted 28 lines
xen/arch/arm/domain_build.c +0
-0if ( iommu_enabled ) dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;xen/arch/arm/include/asm/viommu.h new file +19
-0@@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ +#ifndef __ARCH_ARM_VIOMMU_H__ +#define __ARCH_ARM_VIOMMU_H__ + +#ifdef CONFIG_VIRTUAL_IOMMU + +#include <xen/lib.h> +#include <xen/types.h> +#include <public/xen.h> + +struct viommu_ops { + /* + * Called during domain construction if toolstack requests to enable + * vIOMMU support. + */ + int (*domain_init)(struct domain *d); + + /* + * Called during domain destruction to free resources used by vIOMMU.
xen/arch/arm/include/asm/viommu.h line 29
Quoted 10 lines
xen/arch/arm/include/asm/viommu.h new file +10
-0+*/ +int (*relinquish_resources)(struct domain *d); +}; + +struct viommu_desc { +/* vIOMMU domains init/free operations described above. */ +const struct viommu_ops *ops; + +/* +* ID of vIOMMU. Corresponds to xen_arch_domainconfig.viommu_type.
Did you mean ID rather than type?
xen/arch/arm/include/asm/viommu.h line 32
Quoted 3 lines
xen/arch/arm/include/asm/viommu.h new file +3
-0+* Should be one of XEN_DOMCTL_CONFIG_VIOMMU_xxx +*/ +uint16_t viommu_type;
The domctl is uint8_t.
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH 04/21] xen/arm: vIOMMU: add generic vIOMMU framework
Hi Julien,
Quoted 58 lines · Julien Grall
Hi Rahul,
Quoted 53 lines · Rahul Singh
This patch adds basic framework for vIOMMU.
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/domain.c | 17 +++++++ xen/arch/arm/domain_build.c | 3 ++ xen/arch/arm/include/asm/viommu.h | 70 ++++++++++++++++++++++++++++ xen/drivers/passthrough/Kconfig | 6 +++ xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/viommu.c | 48 +++++++++++++++++++ xen/include/public/arch-arm.h | 4 ++ 7 files changed, 149 insertions(+) create mode 100644 xen/arch/arm/include/asm/viommu.h create mode 100644 xen/drivers/passthrough/arm/viommu.cxen/arch/arm/domain.c +10
-0@@ -37,6 +37,7 @@#include <asm/tee/tee.h> #include <asm/vfp.h> #include <asm/vgic.h> +#include <asm/viommu.h> #include <asm/vtimer.h> #include "vpci.h" @@ -691,6 +692,13 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } + if ( config->arch.viommu_type != XEN_DOMCTL_CONFIG_VIOMMU_NONE ) + { +dprintk (XENLOG_INFO , +"vIOMMU type requested not supported by the platform or Xen\n" ); +return -EINVAL ; + } +return 0 ; } @@ -783,6 +791,9 @@ int arch_domain_create(struct domain *d, if ( (rc = domain_vpci_init(d)) != 0 ) goto fail; + if ( (rc = domain_viommu_init(d, config->arch.viommu_type)) != 0 ) + goto fail; +return 0 ; fail:@@ -998,6 +1009,7 @@ static int relinquish_memory(struct domain *d, struct page_list_head *list)enum { PROG_pci =1 , PROG_tee, + PROG_viommu, PROG_xen, PROG_page, PROG_mapping,@@ -1048,6 +1060,11 @@ int domain_relinquish_resources(struct domain *d)
Quoted 8 lines
xen/arch/arm/domain.c int domain_relinquish_resources(struct domain *d) · line 1066
Quoted 6 lines
xen/arch/arm/domain.c +3
-0if (ret )return ret; + PROGRESS(viommu): + ret =viommu_relinquish_resources (d); +if (ret ) +return ret;I would have expected us to relinquish the vIOMMU resources *before* we detach the devices. So can you explain the ordering?
I think first we need to detach the device that will set the S1 and S2 stage translation to bypass/abort then we need to remove the vIOMMU. Do you have anything that you feel if we relinquish the vIOMMU after detach is not right.
Quoted 15 lines
Quoted 15 lines
xen/arch/arm/domain.c +1
-0+ PROGRESS(xen): ret =relinquish_memory (d, &d->xenpage_list);if ( ret )xen/arch/arm/domain_build.c +1
-0@@ -27,6 +27,7 @@#include <asm/setup.h> #include <asm/cpufeature.h> #include <asm/domain_build.h> +#include <asm/viommu.h> #include <xen/event.h>
Quoted 8 lines
xen/arch/arm/domain_build.c void __init create_domUs(void) · line 3862
Quoted 6 lines
xen/arch/arm/domain_build.c +1
-0#include <xen/irq.h> @@ -3858,6 +3859,7 @@ void __init create_domUs(void)struct domain *d;struct xen_domctl_createdomain d_cfg = { .arch.gic_version =XEN_DOMCTL_CONFIG_GIC_NATIVE , + .arch.viommu_type =viommu_get_type (),I don't think the vIOMMU should be enabled to dom0less domU by default.
I am not enabling the vIOMMU by default. If vIOMMU is disabled via the command line or config option then viommu_get_type() will return XEN_DOMCTL_CONFIG_VIOMMU_NONE and in that case domain_viommu_init() will return 0 without doing anything.
Quoted 43 lines
xen/arch/arm/domain_build.c void __init create_dom0(void) · line 4057
Quoted 9 lines
xen/arch/arm/domain_build.c +1
-0.flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap,/* * The default of 1023 should be sufficient for guests because @@ -4052,6 +4054,7 @@ void __init create_dom0(void)printk (XENLOG_WARNING "Maximum number of vGIC IRQs exceeded.\n" ); dom0_cfg.arch.tee_type =tee_get_type (); dom0_cfg.max_vcpus =dom0_max_vcpus (); + dom0_cfg.arch.viommu_type =viommu_get_type ();Same here.
Quoted 31 lines
xen/arch/arm/domain_build.c +0
-0if ( iommu_enabled ) dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;xen/arch/arm/include/asm/viommu.h new file +23
-0@@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ +#ifndef __ARCH_ARM_VIOMMU_H__ +#define __ARCH_ARM_VIOMMU_H__ + +#ifdef CONFIG_VIRTUAL_IOMMU + +#include <xen/lib.h> +#include <xen/types.h> +#include <public/xen.h> + +struct viommu_ops { +/* +* Called during domain construction if toolstack requests to enable +* vIOMMU support. +*/ +int (*domain_init)(struct domain *d); + +/* +* Called during domain destruction to free resources used by vIOMMU. +*/ +int (*relinquish_resources)(struct domain *d); +}; +
Quoted 8 lines
xen/arch/arm/include/asm/viommu.h line 29
Quoted 6 lines
xen/arch/arm/include/asm/viommu.h new file +6
-0+struct viommu_desc { +/* vIOMMU domains init/free operations described above. */ +const struct viommu_ops *ops; + +/* +* ID of vIOMMU. Corresponds to xen_arch_domainconfig.viommu_type. Did you mean ID rather than type?
I mean here type of physical IOMMU on the host available SMMUv3, SMMUv1 or IPMMU. If you think ID is the right name here I will change it.
Quoted 6 lines
xen/arch/arm/include/asm/viommu.h line 32
Quoted 4 lines
xen/arch/arm/include/asm/viommu.h new file +3
-0+* Should be one of XEN_DOMCTL_CONFIG_VIOMMU_xxx +*/ +uint16_t viommu_type;The domctl is
uint8_t.
Ack. I will fix that.
Regards,
Rahul
Re: [RFC PATCH 04/21] xen/arm: vIOMMU: add generic vIOMMU framework
Quoted 69 lines · Rahul Singh
Hi Julien,
Quoted 58 lines · Julien Grall
Hi Rahul,
Quoted 53 lines · Rahul Singh
This patch adds basic framework for vIOMMU.
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/domain.c | 17 +++++++ xen/arch/arm/domain_build.c | 3 ++ xen/arch/arm/include/asm/viommu.h | 70 ++++++++++++++++++++++++++++ xen/drivers/passthrough/Kconfig | 6 +++ xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/viommu.c | 48 +++++++++++++++++++ xen/include/public/arch-arm.h | 4 ++ 7 files changed, 149 insertions(+)create mode 100644
xen/arch/arm/include/asm/viommu.hcreate mode 100644xen/drivers/passthrough/arm/viommu.cxen/arch/arm/domain.c +10
-0@@ -37,6 +37,7 @@#include <asm/tee/tee.h> #include <asm/vfp.h> #include <asm/vgic.h> +#include <asm/viommu.h> #include <asm/vtimer.h> #include "vpci.h" @@ -691,6 +692,13 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } + if ( config->arch.viommu_type != XEN_DOMCTL_CONFIG_VIOMMU_NONE ) + { +dprintk (XENLOG_INFO , +"vIOMMU type requested not supported by the platform or Xen\n" ); +return -EINVAL ; + } +return 0 ; } @@ -783,6 +791,9 @@ int arch_domain_create(struct domain *d, if ( (rc = domain_vpci_init(d)) != 0 ) goto fail; + if ( (rc = domain_viommu_init(d, config->arch.viommu_type)) != 0 ) + goto fail; +return 0 ; fail:@@ -998,6 +1009,7 @@ static int relinquish_memory(struct domain *d, struct page_list_head *list)enum { PROG_pci =1 , PROG_tee, + PROG_viommu, PROG_xen, PROG_page, PROG_mapping,@@ -1048,6 +1060,11 @@ int domain_relinquish_resources(struct domain *d)Quoted 8 lines
xen/arch/arm/domain.c int domain_relinquish_resources(struct domain *d) · line 1066
Quoted 6 lines
xen/arch/arm/domain.c +3
-0if (ret )return ret; + PROGRESS(viommu): + ret =viommu_relinquish_resources (d); +if (ret ) +return ret;I would have expected us to relinquish the vIOMMU resources *before* we detach the devices. So can you explain the ordering?
Quoted 2 lines
I think first we need to detach the device that will set the S1 and S2 stage translation to bypass/abort then we need to remove the vIOMMU. Do you have anything that you feel if we relinquish the vIOMMU after detach is not right.
iommu_release_dt_devices() will effectively remove the device from the domain. One could decide to store the S1 information per device and therefore it feels wrong that we are removing the S1 information later on.
In addition to that, a device can be removed while the domain is running.
Therefore I think it would make more sense to remove the vIOMMU every time we deassign the device.
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
xen/arch/arm/domain_build.c +1
-0@@ -27,6 +27,7 @@#include <asm/setup.h> #include <asm/cpufeature.h> #include <asm/domain_build.h> +#include <asm/viommu.h> #include <xen/event.h> Quoted 8 lines
xen/arch/arm/domain_build.c void __init create_domUs(void) · line 3862
Quoted 6 lines
xen/arch/arm/domain_build.c +1
-0#include <xen/irq.h> @@ -3858,6 +3859,7 @@ void __init create_domUs(void)struct domain *d;struct xen_domctl_createdomain d_cfg = { .arch.gic_version =XEN_DOMCTL_CONFIG_GIC_NATIVE , + .arch.viommu_type =viommu_get_type (),I don't think the vIOMMU should be enabled to dom0less domU by default.
Quoted 3 lines
I am not enabling the vIOMMU by default. If vIOMMU is disabled via the command line or config option then
viommu_get_type()will returnXEN_DOMCTL_CONFIG_VIOMMU_NONEand in that casedomain_viommu_init()will return 0 without doing anything.
What I mean by default is if the command line is set then you will enable the vIOMMU for both dom0 and dom0less domUs.
vIOMMU should be selected per-domain.
Quoted 53 lines
Quoted 44 lines
xen/arch/arm/domain_build.c void __init create_dom0(void) · line 4057
Quoted 10 lines
xen/arch/arm/domain_build.c +1
-0.flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap,/* * The default of 1023 should be sufficient for guests because @@ -4052,6 +4054,7 @@ void __init create_dom0(void)printk (XENLOG_WARNING "Maximum number of vGIC IRQs exceeded.\n" ); dom0_cfg.arch.tee_type =tee_get_type (); dom0_cfg.max_vcpus =dom0_max_vcpus (); + dom0_cfg.arch.viommu_type =viommu_get_type ();Same here.
Quoted 31 lines
xen/arch/arm/domain_build.c +0
-0if ( iommu_enabled ) dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;xen/arch/arm/include/asm/viommu.h new file +23
-0@@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ +#ifndef __ARCH_ARM_VIOMMU_H__ +#define __ARCH_ARM_VIOMMU_H__ + +#ifdef CONFIG_VIRTUAL_IOMMU + +#include <xen/lib.h> +#include <xen/types.h> +#include <public/xen.h> + +struct viommu_ops { +/* +* Called during domain construction if toolstack requests to enable +* vIOMMU support. +*/ +int (*domain_init)(struct domain *d); + +/* +* Called during domain destruction to free resources used by vIOMMU. +*/ +int (*relinquish_resources)(struct domain *d); +}; +Quoted 8 lines
xen/arch/arm/include/asm/viommu.h line 29
Quoted 6 lines
xen/arch/arm/include/asm/viommu.h new file +6
-0+struct viommu_desc { +/* vIOMMU domains init/free operations described above. */ +const struct viommu_ops *ops; + +/* +* ID of vIOMMU. Corresponds to xen_arch_domainconfig.viommu_type. Did you mean ID rather than type?
Quoted 2 lines
I mean here type of physical IOMMU on the host available SMMUv3, SMMUv1 or IPMMU. If you think ID is the right name here I will change it.
Hmmm... I inverted the two names. I was actually asking whether ID should be Type instead.
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH 04/21] xen/arm: vIOMMU: add generic vIOMMU framework
In reply to Rahul SinghHi Rahul,
Quoted 30 lines · Rahul Singh
This patch adds basic framework for vIOMMU.
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/domain.c | 17 +++++++ xen/arch/arm/domain_build.c | 3 ++ xen/arch/arm/include/asm/viommu.h | 70 ++++++++++++++++++++++++++++ xen/drivers/passthrough/Kconfig | 6 +++ xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/viommu.c | 48 +++++++++++++++++++ xen/include/public/arch-arm.h | 4 ++ 7 files changed, 149 insertions(+) create mode 100644 xen/arch/arm/include/asm/viommu.h create mode 100644 xen/drivers/passthrough/arm/viommu.cxen/arch/arm/domain.c +1
-0@@ -37,6 +37,7 @@#include <asm/tee/tee.h> #include <asm/vfp.h> #include <asm/vgic.h> +#include <asm/viommu.h> #include <asm/vtimer.h> #include "vpci.h"
xen/arch/arm/domain.c int arch_sanitise_domain_config(struct xen_domctl_createdomain *config) · line 698
Quoted 7 lines
xen/arch/arm/domain.c +4
-0@@ -691,6 +692,13 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +if ( config->arch.viommu_type !=XEN_DOMCTL_CONFIG_VIOMMU_NONE ) + { +dprintk (XENLOG_INFO , +"vIOMMU type requested not supported by the platform or Xen\n" );
Maybe a simpler message like for TEE would be better: "Unsupported vIOMMU type"
Quoted 92 lines
xen/arch/arm/domain.c +12
-0+return -EINVAL ; + } +return 0 ; }@@ -783,6 +791,9 @@ int arch_domain_create(struct domain *d,if ( (rc =domain_vpci_init (d)) !=0 ) goto fail; +if ( (rc =domain_viommu_init (d, config->arch.viommu_type)) !=0 ) + goto fail; +return 0 ; fail:@@ -998,6 +1009,7 @@ static int relinquish_memory(struct domain *d, struct page_list_head *list)enum { PROG_pci =1 , PROG_tee, + PROG_viommu, PROG_xen, PROG_page, PROG_mapping,@@ -1048,6 +1060,11 @@ int domain_relinquish_resources(struct domain *d)if (ret )return ret; + PROGRESS(viommu): + ret =viommu_relinquish_resources (d); +if (ret ) +return ret; + PROGRESS(xen): ret =relinquish_memory (d, &d->xenpage_list);if ( ret )xen/arch/arm/domain_build.c +3
-0@@ -27,6 +27,7 @@#include <asm/setup.h> #include <asm/cpufeature.h> #include <asm/domain_build.h> +#include <asm/viommu.h> #include <xen/event.h> #include <xen/irq.h> @@ -3858,6 +3859,7 @@ void __init create_domUs(void)struct domain *d;struct xen_domctl_createdomain d_cfg = { .arch.gic_version =XEN_DOMCTL_CONFIG_GIC_NATIVE , + .arch.viommu_type =viommu_get_type (), .flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap,/* * The default of 1023 should be sufficient for guests because @@ -4052,6 +4054,7 @@ void __init create_dom0(void)printk (XENLOG_WARNING "Maximum number of vGIC IRQs exceeded.\n" ); dom0_cfg.arch.tee_type =tee_get_type (); dom0_cfg.max_vcpus =dom0_max_vcpus (); + dom0_cfg.arch.viommu_type =viommu_get_type ();if ( iommu_enabled ) dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;xen/arch/arm/include/asm/viommu.h new file +22
-0@@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ +#ifndef __ARCH_ARM_VIOMMU_H__ +#define __ARCH_ARM_VIOMMU_H__ + +#ifdef CONFIG_VIRTUAL_IOMMU + +#include <xen/lib.h> +#include <xen/types.h> +#include <public/xen.h> + +struct viommu_ops { +/* +* Called during domain construction if toolstack requests to enable +* vIOMMU support. +*/ +int (*domain_init)(struct domain *d); + +/* +* Called during domain destruction to free resources used by vIOMMU. +*/ +int (*relinquish_resources)(struct domain *d); +};
xen/arch/arm/include/asm/viommu.h line 32
Quoted 10 lines
xen/arch/arm/include/asm/viommu.h new file +10
-0+ +struct viommu_desc { +/* vIOMMU domains init/free operations described above. */ +const struct viommu_ops *ops; + +/* +* ID of vIOMMU. Corresponds to xen_arch_domainconfig.viommu_type. +* Should be one of XEN_DOMCTL_CONFIG_VIOMMU_xxx +*/ +uint16_t viommu_type;
Here the viommu_type is uint16_t ...
xen/arch/arm/include/asm/viommu.h line 41
Quoted 9 lines
xen/arch/arm/include/asm/viommu.h new file +9
-0+}; + +int domain_viommu_init (struct domain *d,uint16_t viommu_type); +int viommu_relinquish_resources (struct domain *d); +uint16_t viommu_get_type (void ); + +#else + +static inline uint8_t viommu_get_type (void )
Here you are returning uint8_t ...
xen/arch/arm/include/asm/viommu.h line 46
Quoted 5 lines
xen/arch/arm/include/asm/viommu.h new file +5
-0+{ +return XEN_DOMCTL_CONFIG_VIOMMU_NONE ; +} + +static inline int domain_viommu_init (struct domain *d,uint16_t viommu_type)
Here you are taking uint16_t. So it looks like that ...
Quoted 109 lines
xen/arch/arm/include/asm/viommu.h new file +24
-0+{ +if (likely (viommu_type ==XEN_DOMCTL_CONFIG_VIOMMU_NONE ) ) +return 0 ; + +return -ENODEV ; +} + +static inline int viommu_relinquish_resources (struct domain *d) +{ +return 0 ; +} + +#endif /* CONFIG_VIRTUAL_IOMMU */ + +#endif /* __ARCH_ARM_VIOMMU_H__ */ + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/ xen/drivers/passthrough/Kconfig +6
-0@@ -35,6 +35,12 @@ config IPMMU_VMSA (H3 ES3.0, M3-W+, etc) or Gen4 SoCs which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is. +config VIRTUAL_IOMMU + bool "Virtual IOMMU Support (UNSUPPORTED)" if UNSUPPORTED + default n + help + Support virtual IOMMU infrastructure to implement vIOMMU. + endif config IOMMU_FORCE_PT_SHARExen/drivers/passthrough/arm/Makefile +1
-0@@ -2,3 +2,4 @@ obj-y += iommu.o iommu_helpers.o iommu_fwspec.o obj-$(CONFIG_ARM_SMMU) += smmu.o obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o obj-$(CONFIG_ARM_SMMU_V3) += smmu-v3.o +obj-$(CONFIG_VIRTUAL_IOMMU) += viommu.oxen/drivers/passthrough/arm/viommu.c new file +48
-0@@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ + +#include <xen/errno.h> +#include <xen/init.h> +#include <xen/types.h> + +#include <asm/viommu.h> + +const struct viommu_desc __read_mostly *cur_viommu; + +int domain_viommu_init (struct domain *d,uint16_t viommu_type) +{ +if ( viommu_type ==XEN_DOMCTL_CONFIG_VIOMMU_NONE ) +return 0 ; + +if ( !cur_viommu ) +return -ENODEV ; + +if ( cur_viommu->viommu_type != viommu_type ) +return -EINVAL ; + +return cur_viommu->ops->domain_init (d); +} + +int viommu_relinquish_resources (struct domain *d) +{ +if ( !cur_viommu ) +return 0 ; + +return cur_viommu->ops->relinquish_resources (d); +} + +uint16_t viommu_get_type (void ) +{ +if ( !cur_viommu ) +return XEN_DOMCTL_CONFIG_VIOMMU_NONE ; + +return cur_viommu->viommu_type; +} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/ xen/include/public/arch-arm.h +0
-0
xen/include/public/arch-arm.h DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t); · line 306
Quoted 10 lines
xen/include/public/arch-arm.h +3
-0@@ -297,10 +297,14 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);#define XEN_DOMCTL_CONFIG_TEE_NONE 0#define XEN_DOMCTL_CONFIG_TEE_OPTEE 1 +#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0 +struct xen_arch_domainconfig {/* IN/OUT */ uint8_t gic_version;/* IN */ +uint8_t viommu_type;
this should be uint16_t and not uint8_t
Quoted 8 lines
xen/include/public/arch-arm.h +1
-1+/* IN */ uint16_t tee_type;/* IN */ uint32_t nr_spis; --2.25.1
~Michal
Re: [RFC PATCH 04/21] xen/arm: vIOMMU: add generic vIOMMU framework
Hi Michal,
Quoted 38 lines · Michal Orzel
Hi Rahul,
Quoted 33 lines · Rahul Singh
This patch adds basic framework for vIOMMU.
Signed-off-by Rahul Singh <[email protected]>---
xen/arch/arm/domain.c | 17 +++++++ xen/arch/arm/domain_build.c | 3 ++ xen/arch/arm/include/asm/viommu.h | 70 ++++++++++++++++++++++++++++ xen/drivers/passthrough/Kconfig | 6 +++ xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/viommu.c | 48 +++++++++++++++++++ xen/include/public/arch-arm.h | 4 ++ 7 files changed, 149 insertions(+)create mode 100644
xen/arch/arm/include/asm/viommu.hcreate mode 100644xen/drivers/passthrough/arm/viommu.cxen/arch/arm/domain.c +1
-0@@ -37,6 +37,7 @@#include <asm/tee/tee.h> #include <asm/vfp.h> #include <asm/vgic.h> +#include <asm/viommu.h> #include <asm/vtimer.h> #include "vpci.h" @@ -691,6 +692,13 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; }
Quoted 5 lines
xen/arch/arm/domain.c int arch_sanitise_domain_config(struct xen_domctl_createdomain *config) · line 698
Quoted 4 lines
xen/arch/arm/domain.c +4
-0+if ( config->arch.viommu_type !=XEN_DOMCTL_CONFIG_VIOMMU_NONE ) + { +dprintk (XENLOG_INFO , +"vIOMMU type requested not supported by the platform or Xen\n" );Maybe a simpler message like for TEE would be better: "Unsupported vIOMMU type"
Ack.
Quoted 235 lines
Quoted 93 lines
xen/arch/arm/domain.c +6
-0+return -EINVAL ; + } +return 0 ; }@@ -783,6 +791,9 @@ int arch_domain_create(struct domain *d,if ( (rc =domain_vpci_init (d)) !=0 ) goto fail; +if ( (rc =domain_viommu_init (d, config->arch.viommu_type)) !=0 ) + goto fail; +return 0 ;fail:
xen/arch/arm/domain.c +6
-0@@ -998,6 +1009,7 @@ static int relinquish_memory(struct domain *d, struct page_list_head *list)enum { PROG_pci =1 , PROG_tee, + PROG_viommu, PROG_xen, PROG_page, PROG_mapping,@@ -1048,6 +1060,11 @@ int domain_relinquish_resources(struct domain *d)if (ret )return ret; + PROGRESS(viommu): + ret =viommu_relinquish_resources (d); +if (ret ) +return ret; + PROGRESS(xen): ret =relinquish_memory (d, &d->xenpage_list);if ( ret )xen/arch/arm/domain_build.c +3
-0@@ -27,6 +27,7 @@#include <asm/setup.h> #include <asm/cpufeature.h> #include <asm/domain_build.h> +#include <asm/viommu.h> #include <xen/event.h> #include <xen/irq.h> @@ -3858,6 +3859,7 @@ void __init create_domUs(void)struct domain *d;struct xen_domctl_createdomain d_cfg = { .arch.gic_version =XEN_DOMCTL_CONFIG_GIC_NATIVE , + .arch.viommu_type =viommu_get_type (), .flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap,/* * The default of 1023 should be sufficient for guests because @@ -4052,6 +4054,7 @@ void __init create_dom0(void)printk (XENLOG_WARNING "Maximum number of vGIC IRQs exceeded.\n" ); dom0_cfg.arch.tee_type =tee_get_type (); dom0_cfg.max_vcpus =dom0_max_vcpus (); + dom0_cfg.arch.viommu_type =viommu_get_type ();if ( iommu_enabled ) dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;xen/arch/arm/include/asm/viommu.h new file +22
-0@@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ +#ifndef __ARCH_ARM_VIOMMU_H__ +#define __ARCH_ARM_VIOMMU_H__ + +#ifdef CONFIG_VIRTUAL_IOMMU + +#include <xen/lib.h> +#include <xen/types.h> +#include <public/xen.h> + +struct viommu_ops { +/* +* Called during domain construction if toolstack requests to enable +* vIOMMU support. +*/ +int (*domain_init)(struct domain *d); + +/* +* Called during domain destruction to free resources used by vIOMMU. +*/ +int (*relinquish_resources)(struct domain *d); +};xen/arch/arm/include/asm/viommu.h line 32
Quoted 10 lines
xen/arch/arm/include/asm/viommu.h new file +10
-0+ +struct viommu_desc { +/* vIOMMU domains init/free operations described above. */ +const struct viommu_ops *ops; + +/* +* ID of vIOMMU. Corresponds to xen_arch_domainconfig.viommu_type. +* Should be one of XEN_DOMCTL_CONFIG_VIOMMU_xxx +*/ +uint16_t viommu_type;Here the
viommu_typeisuint16_t...xen/arch/arm/include/asm/viommu.h line 41
Quoted 9 lines
xen/arch/arm/include/asm/viommu.h new file +9
-0+}; + +int domain_viommu_init (struct domain *d,uint16_t viommu_type); +int viommu_relinquish_resources (struct domain *d); +uint16_t viommu_get_type (void ); + +#else + +static inline uint8_t viommu_get_type (void )Here you are returning
uint8_t...xen/arch/arm/include/asm/viommu.h line 46
Quoted 5 lines
xen/arch/arm/include/asm/viommu.h new file +5
-0+{ +return XEN_DOMCTL_CONFIG_VIOMMU_NONE ; +} + +static inline int domain_viommu_init (struct domain *d,uint16_t viommu_type)Here you are taking
uint16_t. So it looks like that ...Quoted 112 lines
xen/arch/arm/include/asm/viommu.h new file +24
-0+{ +if (likely (viommu_type ==XEN_DOMCTL_CONFIG_VIOMMU_NONE ) ) +return 0 ; + +return -ENODEV ; +} + +static inline int viommu_relinquish_resources (struct domain *d) +{ +return 0 ; +} + +#endif /* CONFIG_VIRTUAL_IOMMU */ + +#endif /* __ARCH_ARM_VIOMMU_H__ */ + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/ xen/drivers/passthrough/Kconfig +6
-0@@ -35,6 +35,12 @@ config IPMMU_VMSA (H3 ES3.0, M3-W+, etc) or Gen4 SoCs which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is. +config VIRTUAL_IOMMU + bool "Virtual IOMMU Support (UNSUPPORTED)" if UNSUPPORTED + default n + help + Support virtual IOMMU infrastructure to implement vIOMMU. +endif
config
IOMMU_FORCE_PT_SHARExen/drivers/passthrough/arm/Makefile +0
-0@@ -2,3 +2,4 @@ obj-y += iommu.o iommu_helpers.o iommu_fwspec.oobj-$(CONFIG_ARM_SMMU) += smmu.o
obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
obj-$(CONFIG_ARM_SMMU_V3) += smmu-v3.oxen/drivers/passthrough/arm/Makefile +1
-0+obj-$(CONFIG_VIRTUAL_IOMMU) += viommu.oxen/drivers/passthrough/arm/viommu.c new file +48
-0@@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ + +#include <xen/errno.h> +#include <xen/init.h> +#include <xen/types.h> + +#include <asm/viommu.h> + +const struct viommu_desc __read_mostly *cur_viommu; + +int domain_viommu_init (struct domain *d,uint16_t viommu_type) +{ +if ( viommu_type ==XEN_DOMCTL_CONFIG_VIOMMU_NONE ) +return 0 ; + +if ( !cur_viommu ) +return -ENODEV ; + +if ( cur_viommu->viommu_type != viommu_type ) +return -EINVAL ; + +return cur_viommu->ops->domain_init (d); +} + +int viommu_relinquish_resources (struct domain *d) +{ +if ( !cur_viommu ) +return 0 ; + +return cur_viommu->ops->relinquish_resources (d); +} + +uint16_t viommu_get_type (void ) +{ +if ( !cur_viommu ) +return XEN_DOMCTL_CONFIG_VIOMMU_NONE ; + +return cur_viommu->viommu_type; +} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/ xen/include/public/arch-arm.h +0
-0@@ -297,10 +297,14 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);#define XEN_DOMCTL_CONFIG_TEE_NONE 0#define XEN_DOMCTL_CONFIG_TEE_OPTEE 1
Quoted 8 lines
xen/include/public/arch-arm.h DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t); · line 306
Quoted 7 lines
xen/include/public/arch-arm.h +3
-0+#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0 +struct xen_arch_domainconfig {/* IN/OUT */ uint8_t gic_version;/* IN */ +uint8_t viommu_type;this should be
uint16_tand notuint8_t
I will modify the in viommu_type to uint8_t in "arch/arm/include/asm/viommu.h" and will also fix everywhere the viommu_type to uint8_t.
Regards,
Rahul
Re: [RFC PATCH 04/21] xen/arm: vIOMMU: add generic vIOMMU framework
Hi Rahul,
Quoted 293 lines · Rahul Singh
Hi Michal,
Quoted 38 lines · Michal Orzel
Hi Rahul,
Quoted 33 lines · Rahul Singh
This patch adds basic framework for vIOMMU.
Signed-off-by Rahul Singh <[email protected]>---
xen/arch/arm/domain.c | 17 +++++++ xen/arch/arm/domain_build.c | 3 ++ xen/arch/arm/include/asm/viommu.h | 70 ++++++++++++++++++++++++++++ xen/drivers/passthrough/Kconfig | 6 +++ xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/viommu.c | 48 +++++++++++++++++++ xen/include/public/arch-arm.h | 4 ++ 7 files changed, 149 insertions(+)create mode 100644
xen/arch/arm/include/asm/viommu.hcreate mode 100644xen/drivers/passthrough/arm/viommu.cxen/arch/arm/domain.c +1
-0@@ -37,6 +37,7 @@#include <asm/tee/tee.h> #include <asm/vfp.h> #include <asm/vgic.h> +#include <asm/viommu.h> #include <asm/vtimer.h> #include "vpci.h" @@ -691,6 +692,13 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; }Quoted 5 lines
xen/arch/arm/domain.c int arch_sanitise_domain_config(struct xen_domctl_createdomain *config) · line 698
Quoted 4 lines
xen/arch/arm/domain.c +4
-0+if ( config->arch.viommu_type !=XEN_DOMCTL_CONFIG_VIOMMU_NONE ) + { +dprintk (XENLOG_INFO , +"vIOMMU type requested not supported by the platform or Xen\n" );Maybe a simpler message like for TEE would be better: "Unsupported vIOMMU type"
Ack.
Quoted 235 lines
Quoted 93 lines
xen/arch/arm/domain.c +6
-0+return -EINVAL ; + } +return 0 ; }@@ -783,6 +791,9 @@ int arch_domain_create(struct domain *d,if ( (rc =domain_vpci_init (d)) !=0 ) goto fail; +if ( (rc =domain_viommu_init (d, config->arch.viommu_type)) !=0 ) + goto fail; +return 0 ;fail:
xen/arch/arm/domain.c +6
-0@@ -998,6 +1009,7 @@ static int relinquish_memory(struct domain *d, struct page_list_head *list)enum { PROG_pci =1 , PROG_tee, + PROG_viommu, PROG_xen, PROG_page, PROG_mapping,@@ -1048,6 +1060,11 @@ int domain_relinquish_resources(struct domain *d)if (ret )return ret; + PROGRESS(viommu): + ret =viommu_relinquish_resources (d); +if (ret ) +return ret; + PROGRESS(xen): ret =relinquish_memory (d, &d->xenpage_list);if ( ret )xen/arch/arm/domain_build.c +3
-0@@ -27,6 +27,7 @@#include <asm/setup.h> #include <asm/cpufeature.h> #include <asm/domain_build.h> +#include <asm/viommu.h> #include <xen/event.h> #include <xen/irq.h> @@ -3858,6 +3859,7 @@ void __init create_domUs(void)struct domain *d;struct xen_domctl_createdomain d_cfg = { .arch.gic_version =XEN_DOMCTL_CONFIG_GIC_NATIVE , + .arch.viommu_type =viommu_get_type (), .flags = XEN_DOMCTL_CDF_hvm | XEN_DOMCTL_CDF_hap,/* * The default of 1023 should be sufficient for guests because @@ -4052,6 +4054,7 @@ void __init create_dom0(void)printk (XENLOG_WARNING "Maximum number of vGIC IRQs exceeded.\n" ); dom0_cfg.arch.tee_type =tee_get_type (); dom0_cfg.max_vcpus =dom0_max_vcpus (); + dom0_cfg.arch.viommu_type =viommu_get_type ();if ( iommu_enabled ) dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;xen/arch/arm/include/asm/viommu.h new file +22
-0@@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ +#ifndef __ARCH_ARM_VIOMMU_H__ +#define __ARCH_ARM_VIOMMU_H__ + +#ifdef CONFIG_VIRTUAL_IOMMU + +#include <xen/lib.h> +#include <xen/types.h> +#include <public/xen.h> + +struct viommu_ops { +/* +* Called during domain construction if toolstack requests to enable +* vIOMMU support. +*/ +int (*domain_init)(struct domain *d); + +/* +* Called during domain destruction to free resources used by vIOMMU. +*/ +int (*relinquish_resources)(struct domain *d); +};xen/arch/arm/include/asm/viommu.h line 32
Quoted 10 lines
xen/arch/arm/include/asm/viommu.h new file +10
-0+ +struct viommu_desc { +/* vIOMMU domains init/free operations described above. */ +const struct viommu_ops *ops; + +/* +* ID of vIOMMU. Corresponds to xen_arch_domainconfig.viommu_type. +* Should be one of XEN_DOMCTL_CONFIG_VIOMMU_xxx +*/ +uint16_t viommu_type;Here the
viommu_typeisuint16_t...xen/arch/arm/include/asm/viommu.h line 41
Quoted 9 lines
xen/arch/arm/include/asm/viommu.h new file +9
-0+}; + +int domain_viommu_init (struct domain *d,uint16_t viommu_type); +int viommu_relinquish_resources (struct domain *d); +uint16_t viommu_get_type (void ); + +#else + +static inline uint8_t viommu_get_type (void )Here you are returning
uint8_t...xen/arch/arm/include/asm/viommu.h line 46
Quoted 5 lines
xen/arch/arm/include/asm/viommu.h new file +5
-0+{ +return XEN_DOMCTL_CONFIG_VIOMMU_NONE ; +} + +static inline int domain_viommu_init (struct domain *d,uint16_t viommu_type)Here you are taking
uint16_t. So it looks like that ...Quoted 112 lines
xen/arch/arm/include/asm/viommu.h new file +24
-0+{ +if (likely (viommu_type ==XEN_DOMCTL_CONFIG_VIOMMU_NONE ) ) +return 0 ; + +return -ENODEV ; +} + +static inline int viommu_relinquish_resources (struct domain *d) +{ +return 0 ; +} + +#endif /* CONFIG_VIRTUAL_IOMMU */ + +#endif /* __ARCH_ARM_VIOMMU_H__ */ + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/ xen/drivers/passthrough/Kconfig +6
-0@@ -35,6 +35,12 @@ config IPMMU_VMSA (H3 ES3.0, M3-W+, etc) or Gen4 SoCs which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is. +config VIRTUAL_IOMMU + bool "Virtual IOMMU Support (UNSUPPORTED)" if UNSUPPORTED + default n + help + Support virtual IOMMU infrastructure to implement vIOMMU. +endif
config
IOMMU_FORCE_PT_SHARExen/drivers/passthrough/arm/Makefile +0
-0@@ -2,3 +2,4 @@ obj-y += iommu.o iommu_helpers.o iommu_fwspec.oobj-$(CONFIG_ARM_SMMU) += smmu.o
obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
obj-$(CONFIG_ARM_SMMU_V3) += smmu-v3.oxen/drivers/passthrough/arm/Makefile +1
-0+obj-$(CONFIG_VIRTUAL_IOMMU) += viommu.oxen/drivers/passthrough/arm/viommu.c new file +48
-0@@ -0,0 +1,48 @@ +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ + +#include <xen/errno.h> +#include <xen/init.h> +#include <xen/types.h> + +#include <asm/viommu.h> + +const struct viommu_desc __read_mostly *cur_viommu; + +int domain_viommu_init (struct domain *d,uint16_t viommu_type) +{ +if ( viommu_type ==XEN_DOMCTL_CONFIG_VIOMMU_NONE ) +return 0 ; + +if ( !cur_viommu ) +return -ENODEV ; + +if ( cur_viommu->viommu_type != viommu_type ) +return -EINVAL ; + +return cur_viommu->ops->domain_init (d); +} + +int viommu_relinquish_resources (struct domain *d) +{ +if ( !cur_viommu ) +return 0 ; + +return cur_viommu->ops->relinquish_resources (d); +} + +uint16_t viommu_get_type (void ) +{ +if ( !cur_viommu ) +return XEN_DOMCTL_CONFIG_VIOMMU_NONE ; + +return cur_viommu->viommu_type; +} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/ xen/include/public/arch-arm.h +0
-0@@ -297,10 +297,14 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);#define XEN_DOMCTL_CONFIG_TEE_NONE 0#define XEN_DOMCTL_CONFIG_TEE_OPTEE 1Quoted 8 lines
xen/include/public/arch-arm.h DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t); · line 306
Quoted 7 lines
xen/include/public/arch-arm.h +3
-0+#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0 +struct xen_arch_domainconfig {/* IN/OUT */ uint8_t gic_version;/* IN */ +uint8_t viommu_type;this should be
uint16_tand notuint8_t
Quoted 2 lines
I will modify the in
viommu_typetouint8_tin "arch/arm/include/asm/viommu.h" and will also fix everywhere theviommu_typetouint8_t.
Also I think that you need to bump XEN_DOMCTL_INTERFACE_VERSION due to the change in struct xen_arch_domainconfig.
Quoted 5 lines
Regards,
Rahul
~Michal
Re: [RFC PATCH 04/21] xen/arm: vIOMMU: add generic vIOMMU framework
Quoted 17 lines · Michal Orzel
Quoted 17 lines
Quoted 16 lines
Quoted 5 lines
xen/include/public/arch-arm.h +0
-0xen/include/public/arch-arm.h DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t); · line 306
Quoted 10 lines
xen/include/public/arch-arm.h +3
-0@@ -297,10 +297,14 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);#define XEN_DOMCTL_CONFIG_TEE_NONE 0#define XEN_DOMCTL_CONFIG_TEE_OPTEE 1 +#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0 +struct xen_arch_domainconfig {/* IN/OUT */ uint8_t gic_version;/* IN */ +uint8_t viommu_type;this should be
uint16_tand notuint8_t
Quoted 4 lines
Quoted 2 lines
I will modify the in
viommu_typetouint8_tin "arch/arm/include/asm/viommu.h" and will also fix everywhere theviommu_typetouint8_t.Also I think that you need to bump
XEN_DOMCTL_INTERFACE_VERSIONdue to the change in structxen_arch_domainconfig.
We only need to bump the domctl version once per release. So if this is the first modification of domctl.h in 4.18 then yes.
That said, I am not sure whether this is necessary here as you are using a padding.
@Rahul, BTW, I think you may need to regenerate the bindings for OCaml and Go.
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH 04/21] xen/arm: vIOMMU: add generic vIOMMU framework
Hi Julien,
Quoted 22 lines · Julien Grall
Quoted 19 lines · Michal Orzel
Quoted 19 lines
Quoted 8 lines
Quoted 8 lines
xen/include/public/arch-arm.h +0
-0@@ -297,10 +297,14 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);#define XEN_DOMCTL_CONFIG_TEE_NONE 0#define XEN_DOMCTL_CONFIG_TEE_OPTEE 1Quoted 8 lines
xen/include/public/arch-arm.h DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t); · line 306
Quoted 7 lines
xen/include/public/arch-arm.h +3
-0+#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0 +struct xen_arch_domainconfig {/* IN/OUT */ uint8_t gic_version;/* IN */ +uint8_t viommu_type;this should be
uint16_tand notuint8_tI will modify the in
viommu_typetouint8_tin "arch/arm/include/asm/viommu.h" and will also fix everywhere theviommu_typetouint8_t.
Quoted 8 lines
Quoted 2 lines
Also I think that you need to bump
XEN_DOMCTL_INTERFACE_VERSIONdue to the change in structxen_arch_domainconfig.We only need to bump the domctl version once per release. So if this is the first modification of
domctl.hin 4.18 then yes.That said, I am not sure whether this is necessary here as you are using a padding.
@Rahul, BTW, I think you may need to regenerate the bindings for OCaml and Go.
Ack. I will check this before sending the v2.
Regards,
Rahul
RFC PATCH 5/21 xen/arm: vsmmuv3: Add dummy support for virtual SMMUv3 for guests
In reply to Rahul Singhdomain_viommu_init() will be called during domain creation and will add the dummy trap handler for virtual IOMMUs for guests.
A host IOMMU list will be created when host IOMMU devices are probed and this list will be used to create the IOMMU device tree node for dom0. For dom0, 1-1 mapping will be established between vIOMMU in dom0 and physical IOMMU.
For domUs, the 1-N mapping will be established between domU and physical IOMMUs. A new area has been reserved in the arm guest physical map at which the emulated vIOMMU node is created in the device tree.
Also set the vIOMMU type to vSMMUv3 to enable vIOMMU framework to call vSMMUv3 domain creation/destroy functions.
--- xen/arch/arm/domain.c | 3 +- xen/arch/arm/include/asm/domain.h | 4 + xen/arch/arm/include/asm/viommu.h | 20 ++++ xen/drivers/passthrough/Kconfig | 8 ++ xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/smmu-v3.c | 7 ++ xen/drivers/passthrough/arm/viommu.c | 30 ++++++ xen/drivers/passthrough/arm/vsmmu-v3.c | 124 +++++++++++++++++++++++++ xen/drivers/passthrough/arm/vsmmu-v3.h | 20 ++++ xen/include/public/arch-arm.h | 7 +- 10 files changed, 222 insertions(+), 2 deletions(-) create mode 100644 xen/drivers/passthrough/arm/vsmmu-v3.c create mode 100644 xen/drivers/passthrough/arm/vsmmu-v3.h
xen/arch/arm/domain.c +2 -1
@@ -692,7 +692,8 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } -if ( config->arch.viommu_type !=XEN_DOMCTL_CONFIG_VIOMMU_NONE ) +if ( config->arch.viommu_type !=XEN_DOMCTL_CONFIG_VIOMMU_NONE && + config->arch.viommu_type !=viommu_get_type () ) {dprintk (XENLOG_INFO ,"vIOMMU type requested not supported by the platform or Xen\n" );
xen/arch/arm/include/asm/domain.h +4 -0
@@ -114,6 +114,10 @@ struct arch_domainvoid *tee;#endif +#ifdef CONFIG_VIRTUAL_IOMMU +struct list_head viommu_list;/* List of virtual IOMMUs */ +#endif + } __cacheline_aligned;struct arch_vcpu
xen/arch/arm/include/asm/viommu.h +20 -0
@@ -5,9 +5,21 @@#ifdef CONFIG_VIRTUAL_IOMMU #include <xen/lib.h> +#include <xen/list.h> #include <xen/types.h> #include <public/xen.h> +extern struct list_head host_iommu_list; + +/* data structure for each hardware IOMMU */ +struct host_iommu { +struct list_head entry; +const struct dt_device_node *dt_node; +paddr_t addr; +paddr_t size; +uint32_t irq; +}; + struct viommu_ops {/* * Called during domain construction if toolstack requests to enable
@@ -35,6 +47,8 @@ struct viommu_desc {int domain_viommu_init (struct domain *d,uint16_t viommu_type);int viommu_relinquish_resources (struct domain *d);uint16_t viommu_get_type (void ); +void add_to_host_iommu_list (paddr_t addr,paddr_t size, +const struct dt_device_node *node);#else
@@ -56,6 +70,12 @@ static inline int viommu_relinquish_resources(struct domain *d)return 0 ; } +static inline void add_to_host_iommu_list (paddr_t addr,paddr_t size, +const struct dt_device_node *node) +{ +return ; +} +#endif /* CONFIG_VIRTUAL_IOMMU */ #endif /* __ARCH_ARM_VIOMMU_H__ */
xen/drivers/passthrough/Kconfig +8 -0
@@ -41,6 +41,14 @@ config VIRTUAL_IOMMU help Support virtual IOMMU infrastructure to implement vIOMMU. +config VIRTUAL_ARM_SMMU_V3 + bool "ARM Ltd. Virtual SMMUv3 Support (UNSUPPORTED)" if UNSUPPORTED + depends on ARM_SMMU_V3 && VIRTUAL_IOMMU + help + Support for implementations of the virtual ARM System MMU architecture + version 3. Virtual SMMUv3 is unsupported feature and should not be used + in production. + endif config IOMMU_FORCE_PT_SHARE
xen/drivers/passthrough/arm/Makefile +1 -0
@@ -3,3 +3,4 @@ obj-$(CONFIG_ARM_SMMU) += smmu.o obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o obj-$(CONFIG_ARM_SMMU_V3) += smmu-v3.o obj-$(CONFIG_VIRTUAL_IOMMU) += viommu.o +obj-$(CONFIG_VIRTUAL_ARM_SMMU_V3) += vsmmu-v3.o
xen/drivers/passthrough/arm/smmu-v3.c +7 -0
@@ -91,6 +91,7 @@#include <asm/platform.h> #include "smmu-v3.h" +#include "vsmmu-v3.h" #define ARM_SMMU_VTCR_SH_IS 3#define ARM_SMMU_VTCR_RGN_WBWA 1
@@ -2680,6 +2681,9 @@ static int arm_smmu_device_probe(struct platform_device *pdev)list_add (&smmu->devices, &arm_smmu_devices);spin_unlock (&arm_smmu_devices_lock); +/* Add to host IOMMU list to initialize vIOMMU for dom0 */ +add_to_host_iommu_list (ioaddr, iosize,dev_to_dt (pdev)); +return 0 ;
@@ -2936,6 +2940,9 @@ static __init int arm_smmu_dt_init(struct dt_device_node *dev,iommu_set_ops (&arm_smmu_iommu_ops); +/* Set vIOMMU type to SMMUv3 */ +vsmmuv3_set_type (); +return 0 ; }
xen/drivers/passthrough/arm/viommu.c +30 -0
@@ -2,12 +2,42 @@#include <xen/errno.h> #include <xen/init.h> +#include <xen/irq.h> #include <xen/types.h> #include <asm/viommu.h> +/* List of all host IOMMUs */ +LIST_HEAD (host_iommu_list); +const struct viommu_desc __read_mostly *cur_viommu; +/* Common function for adding to host_iommu_list */ +void add_to_host_iommu_list (paddr_t addr,paddr_t size, +const struct dt_device_node *node) +{ +struct host_iommu *iommu_data; + + iommu_data = xzalloc(struct host_iommu); +if ( !iommu_data ) +panic ("vIOMMU: Cannot allocate memory for host IOMMU data\n" ); + + iommu_data->addr = addr; + iommu_data->size = size; + iommu_data->dt_node = node; + iommu_data->irq =platform_get_irq (node,0 ); +if ( iommu_data->irq <0 ) + { +gdprintk (XENLOG_ERR , +"vIOMMU: Cannot find a valid IOMMU irq\n" ); +return ; + } + +printk ("vIOMMU: Found IOMMU @0x%" PRIx64"\n" , addr); + +list_add_tail (&iommu_data->entry, &host_iommu_list); +} +int domain_viommu_init (struct domain *d,uint16_t viommu_type) {if ( viommu_type ==XEN_DOMCTL_CONFIG_VIOMMU_NONE )
xen/drivers/passthrough/arm/vsmmu-v3.c new file +124 -0
124 lines not on this page · raw message
xen/drivers/passthrough/arm/vsmmu-v3.h new file +20 -0
@@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ +#ifndef __ARCH_ARM_VSMMU_V3_H__ +#define __ARCH_ARM_VSMMU_V3_H__ + +#include <asm/viommu.h> + +#ifdef CONFIG_VIRTUAL_ARM_SMMU_V3 + +void vsmmuv3_set_type (void ); + +#else + +static inline void vsmmuv3_set_type (void ) +{ +return ; +} + +#endif /* CONFIG_VIRTUAL_ARM_SMMU_V3 */ + +#endif /* __ARCH_ARM_VSMMU_V3_H__ */
xen/include/public/arch-arm.h +6 -1
@@ -297,7 +297,8 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);#define XEN_DOMCTL_CONFIG_TEE_NONE 0#define XEN_DOMCTL_CONFIG_TEE_OPTEE 1 -#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0 +#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0 +#define XEN_DOMCTL_CONFIG_VIOMMU_SMMUV3 1struct xen_arch_domainconfig {/* IN/OUT */
@@ -418,6 +419,10 @@ typedef uint64_t xen_callback_t;#define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000)/* vCPU0..127 */ #define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000) +/* vsmmuv3 ITS mappings */ +#define GUEST_VSMMUV3_BASE xen_mk_ullong(0x04040000) +#define GUEST_VSMMUV3_SIZE xen_mk_ullong(0x00040000) +/* * 256 MB is reserved for VPCI configuration space based on calculation * 256 buses x 32 devices x 8 functions x 4 KB = 256 MB
Signature · 1 line
2.25.1
Re: [RFC PATCH 05/21] xen/arm: vsmmuv3: Add dummy support for virtual SMMUv3 for guests
Hi Rahul,
Quoted 73 lines · Rahul Singh
domain_viommu_init()will be called during domain creation and will add the dummy trap handler for virtual IOMMUs for guests.A host IOMMU list will be created when host IOMMU devices are probed and this list will be used to create the IOMMU device tree node for dom0. For dom0, 1-1 mapping will be established between vIOMMU in dom0 and physical IOMMU.
For domUs, the 1-N mapping will be established between domU and physical IOMMUs. A new area has been reserved in the arm guest physical map at which the emulated vIOMMU node is created in the device tree.
Also set the vIOMMU type to vSMMUv3 to enable vIOMMU framework to call vSMMUv3 domain creation/destroy functions.
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/domain.c | 3 +- xen/arch/arm/include/asm/domain.h | 4 + xen/arch/arm/include/asm/viommu.h | 20 ++++ xen/drivers/passthrough/Kconfig | 8 ++ xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/smmu-v3.c | 7 ++ xen/drivers/passthrough/arm/viommu.c | 30 ++++++ xen/drivers/passthrough/arm/vsmmu-v3.c | 124 +++++++++++++++++++++++++ xen/drivers/passthrough/arm/vsmmu-v3.h | 20 ++++ xen/include/public/arch-arm.h | 7 +- 10 files changed, 222 insertions(+), 2 deletions(-) create mode 100644 xen/drivers/passthrough/arm/vsmmu-v3.c create mode 100644 xen/drivers/passthrough/arm/vsmmu-v3.hxen/arch/arm/domain.c +2
-1@@ -692,7 +692,8 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } -if ( config->arch.viommu_type !=XEN_DOMCTL_CONFIG_VIOMMU_NONE ) +if ( config->arch.viommu_type !=XEN_DOMCTL_CONFIG_VIOMMU_NONE && + config->arch.viommu_type !=viommu_get_type () ) {dprintk (XENLOG_INFO ,"vIOMMU type requested not supported by the platform or Xen\n" );xen/arch/arm/include/asm/domain.h +4
-0@@ -114,6 +114,10 @@ struct arch_domainvoid *tee;#endif +#ifdef CONFIG_VIRTUAL_IOMMU +struct list_head viommu_list;/* List of virtual IOMMUs */ +#endif + } __cacheline_aligned;struct arch_vcpu xen/arch/arm/include/asm/viommu.h +1
-0@@ -5,9 +5,21 @@#ifdef CONFIG_VIRTUAL_IOMMU #include <xen/lib.h> +#include <xen/list.h> #include <xen/types.h> #include <public/xen.h>
xen/arch/arm/include/asm/viommu.h line 20
Quoted 10 lines
xen/arch/arm/include/asm/viommu.h +9
-0+extern struct list_head host_iommu_list; + +/* data structure for each hardware IOMMU */ +struct host_iommu { +struct list_head entry; +const struct dt_device_node *dt_node; +paddr_t addr; +paddr_t size; +uint32_t irq;
You want this to be int and not unsigned. The reason is ...
Quoted 109 lines
xen/arch/arm/include/asm/viommu.h +10
-0+}; +struct viommu_ops {/* * Called during domain construction if toolstack requests to enable @@ -35,6 +47,8 @@ struct viommu_desc {int domain_viommu_init (struct domain *d,uint16_t viommu_type);int viommu_relinquish_resources (struct domain *d);uint16_t viommu_get_type (void ); +void add_to_host_iommu_list (paddr_t addr,paddr_t size, +const struct dt_device_node *node);#else @@ -56,6 +70,12 @@ static inline int viommu_relinquish_resources(struct domain *d)return 0 ; } +static inline void add_to_host_iommu_list (paddr_t addr,paddr_t size, +const struct dt_device_node *node) +{ +return ; +} +#endif /* CONFIG_VIRTUAL_IOMMU */ #endif /* __ARCH_ARM_VIOMMU_H__ */ xen/drivers/passthrough/Kconfig +8
-0@@ -41,6 +41,14 @@ config VIRTUAL_IOMMU help Support virtual IOMMU infrastructure to implement vIOMMU. +config VIRTUAL_ARM_SMMU_V3 + bool "ARM Ltd. Virtual SMMUv3 Support (UNSUPPORTED)" if UNSUPPORTED + depends on ARM_SMMU_V3 && VIRTUAL_IOMMU + help + Support for implementations of the virtual ARM System MMU architecture + version 3. Virtual SMMUv3 is unsupported feature and should not be used + in production. + endif config IOMMU_FORCE_PT_SHARExen/drivers/passthrough/arm/Makefile +1
-0@@ -3,3 +3,4 @@ obj-$(CONFIG_ARM_SMMU) += smmu.o obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o obj-$(CONFIG_ARM_SMMU_V3) += smmu-v3.o obj-$(CONFIG_VIRTUAL_IOMMU) += viommu.o +obj-$(CONFIG_VIRTUAL_ARM_SMMU_V3) += vsmmu-v3.oxen/drivers/passthrough/arm/smmu-v3.c +7
-0@@ -91,6 +91,7 @@#include <asm/platform.h> #include "smmu-v3.h" +#include "vsmmu-v3.h" #define ARM_SMMU_VTCR_SH_IS 3#define ARM_SMMU_VTCR_RGN_WBWA 1@@ -2680,6 +2681,9 @@ static int arm_smmu_device_probe(struct platform_device *pdev)list_add (&smmu->devices, &arm_smmu_devices);spin_unlock (&arm_smmu_devices_lock); +/* Add to host IOMMU list to initialize vIOMMU for dom0 */ +add_to_host_iommu_list (ioaddr, iosize,dev_to_dt (pdev)); +return 0 ;@@ -2936,6 +2940,9 @@ static __init int arm_smmu_dt_init(struct dt_device_node *dev,iommu_set_ops (&arm_smmu_iommu_ops); +/* Set vIOMMU type to SMMUv3 */ +vsmmuv3_set_type (); +return 0 ; }xen/drivers/passthrough/arm/viommu.c +8
-0@@ -2,12 +2,42 @@#include <xen/errno.h> #include <xen/init.h> +#include <xen/irq.h> #include <xen/types.h> #include <asm/viommu.h> +/* List of all host IOMMUs */ +LIST_HEAD (host_iommu_list); +const struct viommu_desc __read_mostly *cur_viommu; +/* Common function for adding to host_iommu_list */ +void add_to_host_iommu_list (paddr_t addr,paddr_t size, +const struct dt_device_node *node) +{
xen/drivers/passthrough/arm/viommu.c line 28
Quoted 10 lines
xen/drivers/passthrough/arm/viommu.c +10
-0+struct host_iommu *iommu_data; + + iommu_data = xzalloc(struct host_iommu); +if ( !iommu_data ) +panic ("vIOMMU: Cannot allocate memory for host IOMMU data\n" ); + + iommu_data->addr = addr; + iommu_data->size = size; + iommu_data->dt_node = node; + iommu_data->irq =platform_get_irq (node,0 );
you are assigning platform_get_irq to it which can return a negative number in case of failure. In your current implementation, the following check does not make sense as unsigned can never be negative.
Quoted 197 lines
xen/drivers/passthrough/arm/viommu.c +12
-0+if ( iommu_data->irq <0 ) + { +gdprintk (XENLOG_ERR , +"vIOMMU: Cannot find a valid IOMMU irq\n" ); +return ; + } + +printk ("vIOMMU: Found IOMMU @0x%" PRIx64"\n" , addr); + +list_add_tail (&iommu_data->entry, &host_iommu_list); +} +int domain_viommu_init (struct domain *d,uint16_t viommu_type) {if ( viommu_type ==XEN_DOMCTL_CONFIG_VIOMMU_NONE )xen/drivers/passthrough/arm/vsmmu-v3.c new file +124
-0@@ -0,0 +1,124 @@ +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ + +#include <xen/param.h> +#include <xen/sched.h> +#include <asm/mmio.h> +#include <asm/viommu.h> + +/* Struct to hold the vIOMMU ops and vIOMMU type */ +extern const struct viommu_desc __read_mostly *cur_viommu; + +struct virt_smmu { +struct domain *d; +struct list_head viommu_list; +}; + +static int vsmmuv3_mmio_write (struct vcpu *v,mmio_info_t *info, +register_t r,void *priv) +{ +return IO_HANDLED ; +} + +static int vsmmuv3_mmio_read (struct vcpu *v,mmio_info_t *info, +register_t *r,void *priv) +{ +return IO_HANDLED ; +} + +static const struct mmio_handler_ops vsmmuv3_mmio_handler = { + .read = vsmmuv3_mmio_read, + .write = vsmmuv3_mmio_write, +}; + +static int vsmmuv3_init_single (struct domain *d,paddr_t addr,paddr_t size) +{ +struct virt_smmu *smmu; + + smmu = xzalloc(struct virt_smmu); +if ( !smmu ) +return -ENOMEM ; + + smmu->d = d; + +register_mmio_handler (d, &vsmmuv3_mmio_handler, addr, size, smmu); + +/* Register the vIOMMU to be able to clean it up later. */ +list_add_tail (&smmu->viommu_list, &d->arch.viommu_list); + +return 0 ; +} + +int domain_vsmmuv3_init (struct domain *d) +{ +int ret; +INIT_LIST_HEAD (&d->arch.viommu_list); + +if (is_hardware_domain (d) ) + { +struct host_iommu *hw_iommu; + +list_for_each_entry (hw_iommu, &host_iommu_list, entry) + { + ret =vsmmuv3_init_single (d, hw_iommu->addr, hw_iommu->size); +if ( ret ) +return ret; + } + } +else + { + ret =vsmmuv3_init_single (d,GUEST_VSMMUV3_BASE ,GUEST_VSMMUV3_SIZE ); +if ( ret ) +return ret; + } + +return 0 ; +} + +int vsmmuv3_relinquish_resources (struct domain *d) +{ +struct virt_smmu *pos, *temp; + +/* Cope with unitialized vIOMMU */ +if (list_head_is_null (&d->arch.viommu_list) ) +return 0 ; + +list_for_each_entry_safe (pos, temp, &d->arch.viommu_list, viommu_list ) + { +list_del (&pos->viommu_list); +xfree (pos); + } + +return 0 ; +} + +static const struct viommu_ops vsmmuv3_ops = { + .domain_init = domain_vsmmuv3_init, + .relinquish_resources = vsmmuv3_relinquish_resources, +}; + +static const struct viommu_desc vsmmuv3_desc = { + .ops = &vsmmuv3_ops, + .viommu_type =XEN_DOMCTL_CONFIG_VIOMMU_SMMUV3 , +}; + +void __init vsmmuv3_set_type (void ) +{ +const struct viommu_desc *desc = &vsmmuv3_desc; + +if ( cur_viommu && (cur_viommu != desc) ) + { +printk ("WARNING: Cannot set vIOMMU, already set to a different value\n" ); +return ; + } + + cur_viommu = desc; +} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/ xen/drivers/passthrough/arm/vsmmu-v3.h new file +20
-0@@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ +#ifndef __ARCH_ARM_VSMMU_V3_H__ +#define __ARCH_ARM_VSMMU_V3_H__ + +#include <asm/viommu.h> + +#ifdef CONFIG_VIRTUAL_ARM_SMMU_V3 + +void vsmmuv3_set_type (void ); + +#else + +static inline void vsmmuv3_set_type (void ) +{ +return ; +} + +#endif /* CONFIG_VIRTUAL_ARM_SMMU_V3 */ + +#endif /* __ARCH_ARM_VSMMU_V3_H__ */ xen/include/public/arch-arm.h +6
-2@@ -297,7 +297,8 @@ DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);#define XEN_DOMCTL_CONFIG_TEE_NONE 0#define XEN_DOMCTL_CONFIG_TEE_OPTEE 1 -#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0 +#define XEN_DOMCTL_CONFIG_VIOMMU_NONE 0 +#define XEN_DOMCTL_CONFIG_VIOMMU_SMMUV3 1struct xen_arch_domainconfig {/* IN/OUT */ @@ -418,6 +419,10 @@ typedef uint64_t xen_callback_t;#define GUEST_GICV3_GICR0_BASE xen_mk_ullong(0x03020000)/* vCPU0..127 */ #define GUEST_GICV3_GICR0_SIZE xen_mk_ullong(0x01000000) +/* vsmmuv3 ITS mappings */ +#define GUEST_VSMMUV3_BASE xen_mk_ullong(0x04040000) +#define GUEST_VSMMUV3_SIZE xen_mk_ullong(0x00040000) +/* * 256 MB is reserved for VPCI configuration space based on calculation * 256 buses x 32 devices x 8 functions x 4 KB = 256 MB --
Quoted 3 lines
2.25.1
~Michal
Re: [RFC PATCH 05/21] xen/arm: vsmmuv3: Add dummy support for virtual SMMUv3 for guests
Hi Michal,
Quoted 195 lines · Michal Orzel
Hi Rahul,
Quoted 73 lines · Rahul Singh
domain_viommu_init()will be called during domain creation and will add the dummy trap handler for virtual IOMMUs for guests.A host IOMMU list will be created when host IOMMU devices are probed and this list will be used to create the IOMMU device tree node for dom0. For dom0, 1-1 mapping will be established between vIOMMU in dom0 and physical IOMMU.
For domUs, the 1-N mapping will be established between domU and physical IOMMUs. A new area has been reserved in the arm guest physical map at which the emulated vIOMMU node is created in the device tree.
Also set the vIOMMU type to vSMMUv3 to enable vIOMMU framework to call vSMMUv3 domain creation/destroy functions.
Signed-off-by Rahul Singh <[email protected]>---
xen/arch/arm/domain.c | 3 +- xen/arch/arm/include/asm/domain.h | 4 + xen/arch/arm/include/asm/viommu.h | 20 ++++ xen/drivers/passthrough/Kconfig | 8 ++ xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/smmu-v3.c | 7 ++ xen/drivers/passthrough/arm/viommu.c | 30 ++++++ xen/drivers/passthrough/arm/vsmmu-v3.c | 124 +++++++++++++++++++++++++ xen/drivers/passthrough/arm/vsmmu-v3.h | 20 ++++ xen/include/public/arch-arm.h | 7 +- 10 files changed, 222 insertions(+), 2 deletions(-)create mode 100644
xen/drivers/passthrough/arm/vsmmu-v3.ccreate mode 100644xen/drivers/passthrough/arm/vsmmu-v3.hxen/arch/arm/domain.c +2
-1@@ -692,7 +692,8 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } -if ( config->arch.viommu_type !=XEN_DOMCTL_CONFIG_VIOMMU_NONE ) +if ( config->arch.viommu_type !=XEN_DOMCTL_CONFIG_VIOMMU_NONE && + config->arch.viommu_type !=viommu_get_type () ) {dprintk (XENLOG_INFO ,"vIOMMU type requested not supported by the platform or Xen\n" );xen/arch/arm/include/asm/domain.h +4
-0@@ -114,6 +114,10 @@ struct arch_domainvoid *tee;#endif +#ifdef CONFIG_VIRTUAL_IOMMU +struct list_head viommu_list;/* List of virtual IOMMUs */ +#endif + } __cacheline_aligned;struct
arch_vcpuxen/arch/arm/include/asm/viommu.h +1
-0@@ -5,9 +5,21 @@#ifdef CONFIG_VIRTUAL_IOMMU #include <xen/lib.h> +#include <xen/list.h> #include <xen/types.h> #include <public/xen.h> xen/arch/arm/include/asm/viommu.h line 20
Quoted 10 lines
xen/arch/arm/include/asm/viommu.h +9
-0+extern struct list_head host_iommu_list; + +/* data structure for each hardware IOMMU */ +struct host_iommu { +struct list_head entry; +const struct dt_device_node *dt_node; +paddr_t addr; +paddr_t size; +uint32_t irq;You want this to be int and not unsigned. The reason is ...
Quoted 105 lines
xen/arch/arm/include/asm/viommu.h +10
-0+}; +struct viommu_ops {/* * Called during domain construction if toolstack requests to enable @@ -35,6 +47,8 @@ struct viommu_desc {int domain_viommu_init (struct domain *d,uint16_t viommu_type);int viommu_relinquish_resources (struct domain *d);uint16_t viommu_get_type (void ); +void add_to_host_iommu_list (paddr_t addr,paddr_t size, +const struct dt_device_node *node);#else @@ -56,6 +70,12 @@ static inline int viommu_relinquish_resources(struct domain *d)return 0 ; } +static inline void add_to_host_iommu_list (paddr_t addr,paddr_t size, +const struct dt_device_node *node) +{ +return ; +} +#endif /* CONFIG_VIRTUAL_IOMMU */ #endif /* __ARCH_ARM_VIOMMU_H__ */ xen/drivers/passthrough/Kconfig +8
-0@@ -41,6 +41,14 @@ config VIRTUAL_IOMMU help Support virtual IOMMU infrastructure to implement vIOMMU. +config VIRTUAL_ARM_SMMU_V3 + bool "ARM Ltd. Virtual SMMUv3 Support (UNSUPPORTED)" if UNSUPPORTED + depends on ARM_SMMU_V3 && VIRTUAL_IOMMU + help + Support for implementations of the virtual ARM System MMU architecture + version 3. Virtual SMMUv3 is unsupported feature and should not be used + in production. +endif
config
IOMMU_FORCE_PT_SHARExen/drivers/passthrough/arm/Makefile +0
-0@@ -3,3 +3,4 @@ obj-$(CONFIG_ARM_SMMU) += smmu.oobj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
obj-$(CONFIG_ARM_SMMU_V3) += smmu-v3.o
obj-$(CONFIG_VIRTUAL_IOMMU) += viommu.oxen/drivers/passthrough/arm/Makefile +1
-0+obj-$(CONFIG_VIRTUAL_ARM_SMMU_V3) += vsmmu-v3.oxen/drivers/passthrough/arm/smmu-v3.c +7
-0@@ -91,6 +91,7 @@#include <asm/platform.h> #include "smmu-v3.h" +#include "vsmmu-v3.h" #define ARM_SMMU_VTCR_SH_IS 3#define ARM_SMMU_VTCR_RGN_WBWA 1@@ -2680,6 +2681,9 @@ static int arm_smmu_device_probe(struct platform_device *pdev)list_add (&smmu->devices, &arm_smmu_devices);spin_unlock (&arm_smmu_devices_lock); +/* Add to host IOMMU list to initialize vIOMMU for dom0 */ +add_to_host_iommu_list (ioaddr, iosize,dev_to_dt (pdev)); +return 0 ;@@ -2936,6 +2940,9 @@ static __init int arm_smmu_dt_init(struct dt_device_node *dev,iommu_set_ops (&arm_smmu_iommu_ops); +/* Set vIOMMU type to SMMUv3 */ +vsmmuv3_set_type (); +return 0 ; }xen/drivers/passthrough/arm/viommu.c +4
-0@@ -2,12 +2,42 @@#include <xen/errno.h> #include <xen/init.h> +#include <xen/irq.h> #include <xen/types.h> #include <asm/viommu.h> +/* List of all host IOMMUs */ +LIST_HEAD (host_iommu_list); +const struct viommu_desc __read_mostly *cur_viommu;
Quoted 16 lines
Quoted 4 lines
xen/drivers/passthrough/arm/viommu.c +4
-0+/* Common function for adding to host_iommu_list */ +void add_to_host_iommu_list (paddr_t addr,paddr_t size, +const struct dt_device_node *node) +{xen/drivers/passthrough/arm/viommu.c line 28
Quoted 10 lines
xen/drivers/passthrough/arm/viommu.c +10
-0+struct host_iommu *iommu_data; + + iommu_data = xzalloc(struct host_iommu); +if ( !iommu_data ) +panic ("vIOMMU: Cannot allocate memory for host IOMMU data\n" ); + + iommu_data->addr = addr; + iommu_data->size = size; + iommu_data->dt_node = node; + iommu_data->irq =platform_get_irq (node,0 );you are assigning
platform_get_irqto it which can return a negative number in case of failure. In your current implementation, the following check does not make sense as unsigned can never be negative.
Ack. I will fix that in next version.
Regards,
Rahul
RFC PATCH 6/21 xen/domctl: Add XEN_DOMCTL_CONFIG_VIOMMU_* and viommu config param
In reply to Rahul SinghAdd new viommu_type field and field values XEN_DOMCTL_CONFIG_VIOMMU_NONE XEN_DOMCTL_CONFIG_VIOMMU_SMMUV3 in xen_arch_domainconfig to enable/disable vIOMMU support for domains.
Also add viommu="N" parameter to xl domain configuration to enable the vIOMMU for the domains. Currently, only the "smmuv3" type is supported for ARM.
--- docs/man/xl.cfg.5.pod.in | 11 +++++++++++ tools/golang/xenlight/helpers.gen.go | 2 ++ tools/golang/xenlight/types.gen.go | 1 + tools/include/libxl.h | 5 +++++ tools/libs/light/libxl_arm.c | 13 +++++++++++++ tools/libs/light/libxl_types.idl | 6 ++++++ tools/xl/xl_parse.c | 9 +++++++++ 7 files changed, 47 insertions(+)
docs/man/xl.cfg.5.pod.in +11 -0
@@ -2870,6 +2870,17 @@ Currently, only the "sbsa_uart" model is supported for ARM. =back +=item B<viommu="N"> + +To enable viommu, user must specify the following option in the VM +config file: + +viommu = "smmuv3" + +Currently, only the "smmuv3" type is supported for ARM. + +=back + =head3 x86 =over 4
tools/golang/xenlight/helpers.gen.go +2 -0
@@ -1117,6 +1117,7 @@ default: return fmt.Errorf("invalid union key '%v'", x.Type)} x.ArchArm.GicVersion = GicVersion(xc.arch_arm.gic_version) x.ArchArm.Vuart = VuartType(xc.arch_arm.vuart) +x.ArchArm.Viommu = ViommuType(xc.arch_arm.viommu) if err := x.ArchX86.MsrRelaxed.fromC(&xc.arch_x86.msr_relaxed);err != nil { return fmt.Errorf("converting field ArchX86.MsrRelaxed: %v", err) }
@@ -1602,6 +1603,7 @@ default: return fmt.Errorf("invalid union key '%v'", x.Type)} xc.arch_arm.gic_version = C.libxl_gic_version(x.ArchArm.GicVersion) xc.arch_arm.vuart = C.libxl_vuart_type(x.ArchArm.Vuart) +xc.arch_arm.viommu = C.libxl_viommu_type(x.ArchArm.Viommu) if err := x.ArchX86.MsrRelaxed.toC(&xc.arch_x86.msr_relaxed); err != nil { return fmt.Errorf("converting field ArchX86.MsrRelaxed: %v", err) }
tools/golang/xenlight/types.gen.go +1 -0
@@ -531,6 +531,7 @@ TypeUnion DomainBuildInfoTypeUnion ArchArm struct { GicVersion GicVersion Vuart VuartType +Viommu ViommuType } ArchX86 struct { MsrRelaxed Defbool
tools/include/libxl.h +5 -0
@@ -278,6 +278,11 @@*/ #define LIBXL_HAVE_BUILDINFO_ARCH_ARM_TEE 1 +/* +* libxl_domain_build_info has the arch_arm.viommu_type field. +*/ +#define LIBXL_HAVE_BUILDINFO_ARM_VIOMMU 1 +/* * LIBXL_HAVE_SOFT_RESET indicates that libxl supports performing * 'soft reset' for domains and there is 'soft_reset' shutdown reason
tools/libs/light/libxl_arm.c +13 -0
@@ -179,6 +179,19 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,return ERROR_FAIL ; } +switch (d_config->b_info.arch_arm.viommu_type) { +case LIBXL_VIOMMU_TYPE_NONE : + config->arch.viommu_type =XEN_DOMCTL_CONFIG_VIOMMU_NONE ; +break ; +case LIBXL_VIOMMU_TYPE_SMMUV3 : + config->arch.viommu_type =XEN_DOMCTL_CONFIG_VIOMMU_SMMUV3 ; +break ; +default : +LOG (ERROR ,"Unknown vIOMMU type %d" , + d_config->b_info.arch_arm.viommu_type); +return ERROR_FAIL ; + } +return 0 ; }
tools/libs/light/libxl_types.idl +6 -0
@@ -492,6 +492,11 @@ libxl_tee_type = Enumeration("tee_type", [ (1, "optee") ], init_val = "LIBXL_TEE_TYPE_NONE") +libxl_viommu_type = Enumeration("viommu_type", [ + (0, "none"), + (1, "smmuv3") + ], init_val = "LIBXL_VIOMMU_TYPE_NONE") + libxl_rdm_reserve = Struct("rdm_reserve", [ ("strategy", libxl_rdm_reserve_strategy), ("policy", libxl_rdm_reserve_policy),
@@ -658,6 +663,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("arch_arm", Struct(None, [("gic_version", libxl_gic_version), ("vuart", libxl_vuart_type), + ("viommu_type", libxl_viommu_type), ])), ("arch_x86", Struct(None, [("msr_relaxed", libxl_defbool), ])),
tools/xl/xl_parse.c +9 -0
@@ -2751,6 +2751,15 @@ skip_usbdev: } } +if (!xlu_cfg_get_string (config,"viommu" , &buf,1 )) { + e =libxl_viommu_type_from_string (buf, &b_info->arch_arm.viommu_type); +if (e) { +fprintf (stderr, +"Unknown vIOMMU type \"%s\" specified\n" , buf); +exit (-ERROR_FAIL ); + } + } +parse_vkb_list (config, d_config); xlu_cfg_get_defbool(config, "xend_suspend_evtchn_compat",
Signature · 1 line
2.25.1
Re: [RFC PATCH 06/21] xen/domctl: Add XEN_DOMCTL_CONFIG_VIOMMU_* and viommu config param
Quoted 9 lines · Rahul Singh
Add new
viommu_typefield and field valuesXEN_DOMCTL_CONFIG_VIOMMU_NONEXEN_DOMCTL_CONFIG_VIOMMU_SMMUV3inxen_arch_domainconfigto enable/disable vIOMMU support for domains.Also add viommu="N" parameter to xl domain configuration to enable the vIOMMU for the domains. Currently, only the "smmuv3" type is supported for ARM.
Signed-off-by Rahul Singh <[email protected]>
Quoted 9 lines
--- docs/man/xl.cfg.5.pod.in | 11 +++++++++++ tools/golang/xenlight/helpers.gen.go | 2 ++ tools/golang/xenlight/types.gen.go | 1 + tools/include/libxl.h | 5 +++++ tools/libs/light/libxl_arm.c | 13 +++++++++++++ tools/libs/light/libxl_types.idl | 6 ++++++ tools/xl/xl_parse.c | 9 +++++++++ 7 files changed, 47 insertions(+)
This diffstat taken together with the title makes me assume that e.g. ...
tools/libs/light/libxl_arm.c int libxl__arch_domain_prepare_config(libxl__gc *gc, · line 184
Quoted 9 lines
tools/libs/light/libxl_arm.c +3
-0@@ -179,6 +179,19 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,return ERROR_FAIL ; } +switch (d_config->b_info.arch_arm.viommu_type) { +case LIBXL_VIOMMU_TYPE_NONE : + config->arch.viommu_type =XEN_DOMCTL_CONFIG_VIOMMU_NONE ;
... this constant doesn't exist yet, and hence this would fail to build at this point in the series. I notice, however, that the constants are introduced in earlier patches. Perhaps the title here wants re-wording?
Jan
Re: [RFC PATCH 06/21] xen/domctl: Add XEN_DOMCTL_CONFIG_VIOMMU_* and viommu config param
Hi Jan,
Quoted 30 lines · Jan Beulich
Quoted 10 lines · Rahul Singh
Add new
viommu_typefield and field valuesXEN_DOMCTL_CONFIG_VIOMMU_NONEXEN_DOMCTL_CONFIG_VIOMMU_SMMUV3inxen_arch_domainconfigto enable/disable vIOMMU support for domains.Also add viommu="N" parameter to xl domain configuration to enable the vIOMMU for the domains. Currently, only the "smmuv3" type is supported for ARM.
Signed-off-by Rahul Singh <[email protected]>---
Quoted 8 lines
docs/man/xl.cfg.5.pod.in | 11 +++++++++++ tools/golang/xenlight/helpers.gen.go | 2 ++ tools/golang/xenlight/types.gen.go | 1 + tools/include/libxl.h | 5 +++++ tools/libs/light/libxl_arm.c | 13 +++++++++++++ tools/libs/light/libxl_types.idl | 6 ++++++ tools/xl/xl_parse.c | 9 +++++++++ 7 files changed, 47 insertions(+)This diffstat taken together with the title makes me assume that e.g. ...
Quoted 6 lines
tools/libs/light/libxl_arm.c +0
-0@@ -179,6 +179,19 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,return ERROR_FAIL ; }
Quoted 7 lines
tools/libs/light/libxl_arm.c int libxl__arch_domain_prepare_config(libxl__gc *gc, · line 184
Quoted 3 lines
tools/libs/light/libxl_arm.c +3
-0+switch (d_config->b_info.arch_arm.viommu_type) { +case LIBXL_VIOMMU_TYPE_NONE : + config->arch.viommu_type =XEN_DOMCTL_CONFIG_VIOMMU_NONE ;... this constant doesn't exist yet, and hence this would fail to build at this point in the series. I notice, however, that the constants are introduced in earlier patches. Perhaps the title here wants re-wording?
Yes, I will fix the commit msg.
Regards,
Rahul
Re: [RFC PATCH 06/21] xen/domctl: Add XEN_DOMCTL_CONFIG_VIOMMU_* and viommu config param
In reply to Rahul SinghHi Rahul,
Quoted 29 lines · Rahul Singh
Add new
viommu_typefield and field valuesXEN_DOMCTL_CONFIG_VIOMMU_NONEXEN_DOMCTL_CONFIG_VIOMMU_SMMUV3inxen_arch_domainconfigto enable/disable vIOMMU support for domains.Also add viommu="N" parameter to xl domain configuration to enable the vIOMMU for the domains. Currently, only the "smmuv3" type is supported for ARM.
Signed-off-by Rahul Singh <[email protected]>--- docs/man/xl.cfg.5.pod.in | 11 +++++++++++ tools/golang/xenlight/helpers.gen.go | 2 ++ tools/golang/xenlight/types.gen.go | 1 + tools/include/libxl.h | 5 +++++ tools/libs/light/libxl_arm.c | 13 +++++++++++++ tools/libs/light/libxl_types.idl | 6 ++++++ tools/xl/xl_parse.c | 9 +++++++++ 7 files changed, 47 insertions(+)docs/man/xl.cfg.5.pod.in +0
-0@@ -2870,6 +2870,17 @@ Currently, only the "sbsa_uart" model is supported for ARM. =back
docs/man/xl.cfg.5.pod.in Currently, only the "sbsa_uart" model is supported for ARM. · line 2882
Quoted 10 lines
docs/man/xl.cfg.5.pod.in +10
-0+=item B<viommu="N"> + +To enable viommu, user must specify the following option in the VM +config file: + +viommu = "smmuv3" + +Currently, only the "smmuv3" type is supported for ARM. + +=back
You need to remove this "=back" or the one before your new section, because currently the build of docs fails with error:
man/xl.cfg.5.pod around line 2873: '=item' outside of any '=over'
~Michal
RFC PATCH 7/21 xen/arm: vIOMMU: Add cmdline boot option "viommu = <boolean>"
In reply to Rahul SinghAdd cmdline boot option "viommu = <boolean>" to enable or disable the virtual iommu support for guests on ARM.
--- docs/misc/xen-command-line.pandoc | 7 +++++++ xen/arch/arm/include/asm/viommu.h | 11 +++++++++++ xen/drivers/passthrough/arm/viommu.c | 9 +++++++++ xen/drivers/passthrough/arm/vsmmu-v3.c | 3 +++ 4 files changed, 30 insertions(+)
docs/misc/xen-command-line.pandoc +7 -0
@@ -1896,6 +1896,13 @@ This option can be specified more than once (up to 8 times at present). Flag to enable or disable support for PCI passthrough +### viommu (arm) +> `= <boolean>` + +> Default: `false` + +Flag to enable or disable support for Virtual IOMMU for guests. + ### pcid (x86) > `= <boolean> | xpti=<bool>`
xen/arch/arm/include/asm/viommu.h +11 -0
@@ -10,6 +10,7 @@#include <public/xen.h> extern struct list_head host_iommu_list; +extern bool viommu_enabled;/* data structure for each hardware IOMMU */ struct host_iommu {
@@ -50,6 +51,11 @@ uint16_t viommu_get_type(void);void add_to_host_iommu_list (paddr_t addr,paddr_t size,const struct dt_device_node *node); +static always_inline bool is_viommu_enabled(void) +{ +return viommu_enabled; +} +#else static inline uint8_t viommu_get_type (void )
@@ -76,6 +82,11 @@ static inline void add_to_host_iommu_list(paddr_t addr, paddr_t size,return ; } +static always_inline bool is_viommu_enabled(void) +{ +return false; +} +#endif /* CONFIG_VIRTUAL_IOMMU */ #endif /* __ARCH_ARM_VIOMMU_H__ */
xen/drivers/passthrough/arm/viommu.c +9 -0
@@ -3,6 +3,7 @@#include <xen/errno.h> #include <xen/init.h> #include <xen/irq.h> +#include <xen/param.h> #include <xen/types.h> #include <asm/viommu.h>
@@ -38,8 +39,16 @@ void add_to_host_iommu_list(paddr_t addr, paddr_t size,list_add_tail (&iommu_data->entry, &host_iommu_list); } +/* By default viommu is disabled. */ +bool __read_mostly viommu_enabled; +boolean_param ("viommu" , viommu_enabled); +int domain_viommu_init (struct domain *d,uint16_t viommu_type) { +/* Enable viommu when it has been enabled explicitly (viommu=on). */ +if ( !viommu_enabled ) +return 0 ; +if ( viommu_type ==XEN_DOMCTL_CONFIG_VIOMMU_NONE )return 0 ;
xen/drivers/passthrough/arm/vsmmu-v3.c +3 -0
@@ -105,6 +105,9 @@ void __init vsmmuv3_set_type(void) {const struct viommu_desc *desc = &vsmmuv3_desc; +if ( !is_viommu_enabled () ) +return ; +if ( cur_viommu && (cur_viommu != desc) ) {printk ("WARNING: Cannot set vIOMMU, already set to a different value\n" );
Signature · 1 line
2.25.1
Re: [RFC PATCH 07/21] xen/arm: vIOMMU: Add cmdline boot option "viommu = <boolean>"
Quoted 5 lines · Rahul Singh
docs/misc/xen-command-line.pandoc +0
-0@@ -1896,6 +1896,13 @@ This option can be specified more than once (up to 8 times at present). Flag to enable or disable support for PCI passthrough
docs/misc/xen-command-line.pandoc This option can be specified more than once (up to 8 times at present). · line 1907
Quoted 10 lines
docs/misc/xen-command-line.pandoc +7
-0+### viommu (arm) +> `= <boolean>` + +> Default: `false` + +Flag to enable or disable support for Virtual IOMMU for guests. + ### pcid (x86) > `= <boolean> | xpti=<bool>`
I'm sorry for being blunt, but how did you select this particular slot for insertion, when the whole file clearly at least tries to have options sorted alphabetically.
Apart from that I wonder whether this actually needs a command line control, when supposedly this is controlled on a per-domain basis anyway.
Jan
RFC PATCH 8/21 xen/arm: vsmmuv3: Add support for registers emulation
In reply to Rahul SinghAdd initial support for various emulated registers for virtual SMMUv3 for guests and also add support for virtual cmdq and eventq.
--- xen/drivers/passthrough/arm/vsmmu-v3.c | 281 +++++++++++++++++++++++++ 1 file changed, 281 insertions(+)
xen/drivers/passthrough/arm/vsmmu-v3.c +281 -0
@@ -3,25 +3,302 @@#include <xen/param.h> #include <xen/sched.h> #include <asm/mmio.h> +#include <asm/vgic-emul.h> #include <asm/viommu.h> +#include <asm/vreg.h> + +#include "smmu-v3.h" + +/* Register Definition */ +#define ARM_SMMU_IDR2 0x8 +#define ARM_SMMU_IDR3 0xc +#define ARM_SMMU_IDR4 0x10 +#define IDR0_TERM_MODEL (1 << 26) +#define IDR3_RIL (1 << 10) +#define CR0_RESERVED 0xFFFFFC20 +#define SMMU_IDR1_SIDSIZE 16 +#define SMMU_CMDQS 19 +#define SMMU_EVTQS 19 +#define DWORDS_BYTES 8/* Struct to hold the vIOMMU ops and vIOMMU type */ extern const struct viommu_desc __read_mostly *cur_viommu; +/* virtual smmu queue */ +struct arm_vsmmu_queue { +uint64_t q_base;/* base register */ +uint32_t prod; +uint32_t cons; +uint8_t ent_size; +uint8_t max_n_shift; +}; +struct virt_smmu {struct domain *d;struct list_head viommu_list; +uint8_t sid_split; +uint32_t features; +uint32_t cr[3 ]; +uint32_t cr0ack; +uint32_t gerror; +uint32_t gerrorn; +uint32_t strtab_base_cfg; +uint64_t strtab_base; +uint32_t irq_ctrl; +uint64_t gerror_irq_cfg0; +uint64_t evtq_irq_cfg0; +struct arm_vsmmu_queue evtq, cmdq; };static int vsmmuv3_mmio_write (struct vcpu *v,mmio_info_t *info,register_t r,void *priv) { +struct virt_smmu *smmu = priv; +uint64_t reg; +uint32_t reg32; + +switch ( info->gpa &0xffff ) + { +case VREG32 (ARM_SMMU_CR0 ): + reg32 = smmu->cr[0 ]; +vreg_reg32_update (®32, r, info); + smmu->cr[0 ] = reg32; + smmu->cr0ack = reg32 & ~CR0_RESERVED ; +break ; + +case VREG32 (ARM_SMMU_CR1 ): + reg32 = smmu->cr[1 ]; +vreg_reg32_update (®32, r, info); + smmu->cr[1 ] = reg32; +break ; + +case VREG32 (ARM_SMMU_CR2 ): + reg32 = smmu->cr[2 ]; +vreg_reg32_update (®32, r, info); + smmu->cr[2 ] = reg32; +break ; + +case VREG64 (ARM_SMMU_STRTAB_BASE ): + reg = smmu->strtab_base; +vreg_reg64_update (®, r, info); + smmu->strtab_base = reg; +break ; + +case VREG32 (ARM_SMMU_STRTAB_BASE_CFG ): + reg32 = smmu->strtab_base_cfg; +vreg_reg32_update (®32, r, info); + smmu->strtab_base_cfg = reg32; + + smmu->sid_split =FIELD_GET (STRTAB_BASE_CFG_SPLIT , reg32); + smmu->features |=STRTAB_BASE_CFG_FMT_2LVL ; +break ; + +case VREG32 (ARM_SMMU_CMDQ_BASE ): + reg = smmu->cmdq.q_base; +vreg_reg64_update (®, r, info); + smmu->cmdq.q_base = reg; + smmu->cmdq.max_n_shift =FIELD_GET (Q_BASE_LOG2SIZE , smmu->cmdq.q_base); +if ( smmu->cmdq.max_n_shift >SMMU_CMDQS ) + smmu->cmdq.max_n_shift =SMMU_CMDQS ; +break ; + +case VREG32 (ARM_SMMU_CMDQ_PROD ): + reg32 = smmu->cmdq.prod; +vreg_reg32_update (®32, r, info); + smmu->cmdq.prod = reg32; +break ; + +case VREG32 (ARM_SMMU_CMDQ_CONS ): + reg32 = smmu->cmdq.cons; +vreg_reg32_update (®32, r, info); + smmu->cmdq.cons = reg32; +break ; + +case VREG32 (ARM_SMMU_EVTQ_BASE ): + reg = smmu->evtq.q_base; +vreg_reg64_update (®, r, info); + smmu->evtq.q_base = reg; + smmu->evtq.max_n_shift =FIELD_GET (Q_BASE_LOG2SIZE , smmu->evtq.q_base); +if ( smmu->cmdq.max_n_shift >SMMU_EVTQS ) + smmu->cmdq.max_n_shift =SMMU_EVTQS ; +break ; + +case VREG32 (ARM_SMMU_EVTQ_PROD ): + reg32 = smmu->evtq.prod; +vreg_reg32_update (®32, r, info); + smmu->evtq.prod = reg32; +break ; + +case VREG32 (ARM_SMMU_EVTQ_CONS ): + reg32 = smmu->evtq.cons; +vreg_reg32_update (®32, r, info); + smmu->evtq.cons = reg32; +break ; + +case VREG32 (ARM_SMMU_IRQ_CTRL ): + reg32 = smmu->irq_ctrl; +vreg_reg32_update (®32, r, info); + smmu->irq_ctrl = reg32; +break ; + +case VREG64 (ARM_SMMU_GERROR_IRQ_CFG0 ): + reg = smmu->gerror_irq_cfg0; +vreg_reg64_update (®, r, info); + smmu->gerror_irq_cfg0 = reg; +break ; + +case VREG64 (ARM_SMMU_EVTQ_IRQ_CFG0 ): + reg = smmu->evtq_irq_cfg0; +vreg_reg64_update (®, r, info); + smmu->evtq_irq_cfg0 = reg; +break ; + +case VREG32 (ARM_SMMU_GERRORN ): + reg = smmu->gerrorn; +vreg_reg64_update (®, r, info); + smmu->gerrorn = reg; +break ; + +default : +printk (XENLOG_G_ERR +"%pv: vSMMUv3: unhandled write r%d offset %" PRIpaddr"\n" , + v, info->dabt.reg, (unsigned long )info->gpa &0xffff ); +return IO_ABORT ; + } +return IO_HANDLED ; }static int vsmmuv3_mmio_read (struct vcpu *v,mmio_info_t *info,register_t *r,void *priv) { +struct virt_smmu *smmu = priv; +uint64_t reg; + +switch ( info->gpa &0xffff ) + { +case VREG32 (ARM_SMMU_IDR0 ): + reg =FIELD_PREP (IDR0_S1P ,1 ) |FIELD_PREP (IDR0_TTF ,2 ) | +FIELD_PREP (IDR0_COHACC ,0 ) |FIELD_PREP (IDR0_ASID16 ,1 ) | +FIELD_PREP (IDR0_TTENDIAN ,0 ) |FIELD_PREP (IDR0_STALL_MODEL ,1 ) | +FIELD_PREP (IDR0_ST_LVL ,1 ) |FIELD_PREP (IDR0_TERM_MODEL ,1 ); + *r =vreg_reg32_extract (reg, info); +break ; + +case VREG32 (ARM_SMMU_IDR1 ): + reg =FIELD_PREP (IDR1_SIDSIZE ,SMMU_IDR1_SIDSIZE ) | +FIELD_PREP (IDR1_CMDQS ,SMMU_CMDQS ) | +FIELD_PREP (IDR1_EVTQS ,SMMU_EVTQS ); + *r =vreg_reg32_extract (reg, info); +break ; + +case VREG32 (ARM_SMMU_IDR2 ): + goto read_reserved; + +case VREG32 (ARM_SMMU_IDR3 ): + reg =FIELD_PREP (IDR3_RIL ,0 ); + *r =vreg_reg32_extract (reg, info); +break ; + +case VREG32 (ARM_SMMU_IDR4 ): + goto read_impl_defined; + +case VREG32 (ARM_SMMU_IDR5 ): + reg =FIELD_PREP (IDR5_GRAN4K ,1 ) |FIELD_PREP (IDR5_GRAN16K ,1 ) | +FIELD_PREP (IDR5_GRAN64K ,1 ) |FIELD_PREP (IDR5_OAS ,IDR5_OAS_48_BIT ); + *r =vreg_reg32_extract (reg, info); +break ; + +case VREG32 (ARM_SMMU_CR0 ): + *r =vreg_reg32_extract (smmu->cr[0 ], info); +break ; + +case VREG32 (ARM_SMMU_CR0ACK ): + *r =vreg_reg32_extract (smmu->cr0ack, info); +break ; + +case VREG32 (ARM_SMMU_CR1 ): + *r =vreg_reg32_extract (smmu->cr[1 ], info); +break ; + +case VREG32 (ARM_SMMU_CR2 ): + *r =vreg_reg32_extract (smmu->cr[2 ], info); +break ; + +case VREG32 (ARM_SMMU_STRTAB_BASE ): + *r =vreg_reg64_extract (smmu->strtab_base, info); +break ; + +case VREG32 (ARM_SMMU_STRTAB_BASE_CFG ): + *r =vreg_reg32_extract (smmu->strtab_base_cfg, info); +break ; + +case VREG32 (ARM_SMMU_CMDQ_BASE ): + *r =vreg_reg64_extract (smmu->cmdq.q_base, info); +break ; + +case VREG32 (ARM_SMMU_CMDQ_PROD ): + *r =vreg_reg32_extract (smmu->cmdq.prod, info); +break ; + +case VREG32 (ARM_SMMU_CMDQ_CONS ): + *r =vreg_reg32_extract (smmu->cmdq.cons, info); +break ; + +case VREG32 (ARM_SMMU_EVTQ_BASE ): + *r =vreg_reg64_extract (smmu->evtq.q_base, info); +break ; + +case VREG32 (ARM_SMMU_EVTQ_PROD ): + *r =vreg_reg32_extract (smmu->evtq.prod, info); +break ; + +case VREG32 (ARM_SMMU_EVTQ_CONS ): + *r =vreg_reg32_extract (smmu->evtq.cons, info); +break ; + +case VREG32 (ARM_SMMU_IRQ_CTRL ): +case VREG32 (ARM_SMMU_IRQ_CTRLACK ): + *r =vreg_reg32_extract (smmu->irq_ctrl, info); +break ; + +case VREG64 (ARM_SMMU_GERROR_IRQ_CFG0 ): + *r =vreg_reg64_extract (smmu->gerror_irq_cfg0, info); +break ; + +case VREG64 (ARM_SMMU_EVTQ_IRQ_CFG0 ): + *r =vreg_reg64_extract (smmu->evtq_irq_cfg0, info); +break ; + +case VREG32 (ARM_SMMU_GERROR ): + *r =vreg_reg64_extract (smmu->gerror, info); +break ; + +case VREG32 (ARM_SMMU_GERRORN ): + *r =vreg_reg64_extract (smmu->gerrorn, info); +break ; + +default : +printk (XENLOG_G_ERR +"%pv: vSMMUv3: unhandled read r%d offset %" PRIpaddr"\n" , + v, info->dabt.reg, (unsigned long )info->gpa &0xffff ); +return IO_ABORT ; + } + +return IO_HANDLED ; + + read_impl_defined: +printk (XENLOG_G_DEBUG +"%pv: vSMMUv3: RAZ on implementation defined register offset %" PRIpaddr"\n" , + v, info->gpa &0xffff ); + *r =0 ; +return IO_HANDLED ; + + read_reserved: +printk (XENLOG_G_DEBUG +"%pv: vSMMUv3: RAZ on reserved register offset %" PRIpaddr"\n" , + v, info->gpa &0xffff ); + *r =0 ;return IO_HANDLED ; }
@@ -39,6 +316,10 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size)return -ENOMEM ; smmu->d = d; + smmu->cmdq.q_base =FIELD_PREP (Q_BASE_LOG2SIZE ,SMMU_CMDQS ); + smmu->cmdq.ent_size =CMDQ_ENT_DWORDS *DWORDS_BYTES ; + smmu->evtq.q_base =FIELD_PREP (Q_BASE_LOG2SIZE ,SMMU_EVTQS ); + smmu->evtq.ent_size =EVTQ_ENT_DWORDS *DWORDS_BYTES ;register_mmio_handler (d, &vsmmuv3_mmio_handler, addr, size, smmu);
Signature · 1 line
2.25.1
Re: [RFC PATCH 08/21] xen/arm: vsmmuv3: Add support for registers emulation
Hi Rahul,
I have only skimmed through the patch so far.
xen/drivers/passthrough/arm/vsmmu-v3.c line 62
Quoted 10 lines · Rahul Singh
static int vsmmuv3_mmio_write(struct vcpu *v,
mmio_info_t*info,
register_tr, void *priv)
{xen/drivers/passthrough/arm/vsmmu-v3.c +7
-0+struct virt_smmu *smmu = priv; +uint64_t reg; +uint32_t reg32; + +switch ( info->gpa &0xffff ) + { +case VREG32 (ARM_SMMU_CR0 ):
Shouldn't this code (and all the other register emulations) be protected for concurrent access in some way?
xen/drivers/passthrough/arm/vsmmu-v3.c line 66
Quoted 4 lines
xen/drivers/passthrough/arm/vsmmu-v3.c +4
-0+ reg32 = smmu->cr[0 ]; +vreg_reg32_update (®32, r, info); + smmu->cr[0 ] = reg32; + smmu->cr0ack = reg32 & ~CR0_RESERVED ;
Looking at the use. I think it doesn't look necessary to have a copy of cr0 with just the reserved bit(s) unset. Instead, it would be better to clear the bit(s) when reading it.
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH 08/21] xen/arm: vsmmuv3: Add support for registers emulation
Hi Julien,
Quoted 19 lines · Julien Grall
Hi Rahul,
I have only skimmed through the patch so far.
xen/drivers/passthrough/arm/vsmmu-v3.c line 62
Quoted 10 lines · Rahul Singh
static int vsmmuv3_mmio_write(struct vcpu *v,
mmio_info_t*info,
register_tr, void *priv)
{xen/drivers/passthrough/arm/vsmmu-v3.c +7
-0+struct virt_smmu *smmu = priv; +uint64_t reg; +uint32_t reg32; + +switch ( info->gpa &0xffff ) + { +case VREG32 (ARM_SMMU_CR0 ):
Quoted 1 line
Shouldn't this code (and all the other register emulations) be protected for concurrent access in some way?
Yes, I agree I will add the lock for register emulations in next v2.
Quoted 8 lines
xen/drivers/passthrough/arm/vsmmu-v3.c line 66
Quoted 6 lines
xen/drivers/passthrough/arm/vsmmu-v3.c +4
-0+ reg32 = smmu->cr[0 ]; +vreg_reg32_update (®32, r, info); + smmu->cr[0 ] = reg32; + smmu->cr0ack = reg32 & ~CR0_RESERVED ;Looking at the use. I think it doesn't look necessary to have a copy of cr0 with just the reserved bit(s) unset. Instead, it would be better to clear the bit(s) when reading it.
Ack.
Regards,
Rahul
RFC PATCH 9/21 xen/arm: vsmmuv3: Add support for cmdqueue handling
In reply to Rahul SinghAdd support for virtual cmdqueue handling for guests
--- xen/drivers/passthrough/arm/vsmmu-v3.c | 101 +++++++++++++++++++++++++ 1 file changed, 101 insertions(+)
xen/drivers/passthrough/arm/vsmmu-v3.c +101 -0
@@ -1,5 +1,6 @@/* SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause) */ +#include <xen/guest_access.h> #include <xen/param.h> #include <xen/sched.h> #include <asm/mmio.h>
@@ -24,6 +25,26 @@/* Struct to hold the vIOMMU ops and vIOMMU type */ extern const struct viommu_desc __read_mostly *cur_viommu; +/* SMMUv3 command definitions */ +#define CMDQ_OP_PREFETCH_CFG 0x1 +#define CMDQ_OP_CFGI_STE 0x3 +#define CMDQ_OP_CFGI_ALL 0x4 +#define CMDQ_OP_CFGI_CD 0x5 +#define CMDQ_OP_CFGI_CD_ALL 0x6 +#define CMDQ_OP_TLBI_NH_ASID 0x11 +#define CMDQ_OP_TLBI_NH_VA 0x12 +#define CMDQ_OP_TLBI_NSNH_ALL 0x30 +#define CMDQ_OP_CMD_SYNC 0x46 + +/* Queue Handling */ +#define Q_BASE (q) ((q)->q_base & Q_BASE_ADDR_MASK) +#define Q_CONS_ENT (q) (Q_BASE(q) + Q_IDX(q, (q)->cons) * (q)->ent_size) +#define Q_PROD_ENT (q) (Q_BASE(q) + Q_IDX(q, (q)->prod) * (q)->ent_size) + +/* Helper Macros */ +#define smmu_get_cmdq_enabled (x) FIELD_GET(CR0_CMDQEN, x) +#define smmu_cmd_get_command (x) FIELD_GET(CMDQ_0_OP, x) +/* virtual smmu queue */ struct arm_vsmmu_queue {uint64_t q_base;/* base register */
@@ -48,8 +69,80 @@ struct virt_smmu {uint64_t gerror_irq_cfg0;uint64_t evtq_irq_cfg0;struct arm_vsmmu_queue evtq, cmdq; +spinlock_t cmd_queue_lock; }; +/* Queue manipulation functions */ +static bool queue_empty (struct arm_vsmmu_queue *q) +{ +return Q_IDX (q, q->prod) ==Q_IDX (q, q->cons) && +Q_WRP (q, q->prod) ==Q_WRP (q, q->cons); +} + +static void queue_inc_cons (struct arm_vsmmu_queue *q) +{ +uint32_t cons = (Q_WRP (q, q->cons) |Q_IDX (q, q->cons)) +1 ; + q->cons =Q_OVF (q->cons) |Q_WRP (q, cons) |Q_IDX (q, cons); +} + +static void dump_smmu_command (uint64_t *command) +{ +gdprintk (XENLOG_ERR ,"cmd 0x%02llx: %016lx %016lx\n" , +smmu_cmd_get_command (command[0 ]), command[0 ], command[1 ]); +} +static int arm_vsmmu_handle_cmds (struct virt_smmu *smmu) +{ +struct arm_vsmmu_queue *q = &smmu->cmdq; +struct domain *d = smmu->d; +uint64_t command[CMDQ_ENT_DWORDS ]; +paddr_t addr; + +if ( !smmu_get_cmdq_enabled (smmu->cr[0 ]) ) +return 0 ; + +while ( !queue_empty (q) ) + { +int ret; + + addr =Q_CONS_ENT (q); + ret =access_guest_memory_by_ipa (d, addr, command, +sizeof (command), false); +if ( ret ) +return ret; + +switch (smmu_cmd_get_command (command[0 ]) ) + { +case CMDQ_OP_CFGI_STE : +break ; +case CMDQ_OP_PREFETCH_CFG : +case CMDQ_OP_CFGI_CD : +case CMDQ_OP_CFGI_CD_ALL : +case CMDQ_OP_CFGI_ALL : +case CMDQ_OP_CMD_SYNC : +break ; +case CMDQ_OP_TLBI_NH_ASID : +case CMDQ_OP_TLBI_NSNH_ALL : +case CMDQ_OP_TLBI_NH_VA : +if ( !iommu_iotlb_flush_all (smmu->d,1 ) ) +break ; +default : +gdprintk (XENLOG_ERR ,"vSMMUv3: unhandled command\n" ); +dump_smmu_command (command); +break ; + } + +if ( ret ) + { +gdprintk (XENLOG_ERR , +"vSMMUv3: command error %d while handling command\n" , + ret); +dump_smmu_command (command); + } +queue_inc_cons (q); + } +return 0 ; +} +static int vsmmuv3_mmio_write (struct vcpu *v,mmio_info_t *info,register_t r,void *priv) {
@@ -103,9 +196,15 @@ static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,break ;case VREG32 (ARM_SMMU_CMDQ_PROD ): +spin_lock (&smmu->cmd_queue_lock); reg32 = smmu->cmdq.prod;vreg_reg32_update (®32, r, info); smmu->cmdq.prod = reg32; + +if (arm_vsmmu_handle_cmds (smmu) ) +gdprintk (XENLOG_ERR ,"error handling vSMMUv3 commands\n" ); + +spin_unlock (&smmu->cmd_queue_lock);break ;case VREG32 (ARM_SMMU_CMDQ_CONS ):
@@ -321,6 +420,8 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr, paddr_t size) smmu->evtq.q_base =FIELD_PREP (Q_BASE_LOG2SIZE ,SMMU_EVTQS ); smmu->evtq.ent_size =EVTQ_ENT_DWORDS *DWORDS_BYTES ; +spin_lock_init (&smmu->cmd_queue_lock); +register_mmio_handler (d, &vsmmuv3_mmio_handler, addr, size, smmu);/* Register the vIOMMU to be able to clean it up later. */
Signature · 1 line
2.25.1
Re: [RFC PATCH 09/21] xen/arm: vsmmuv3: Add support for cmdqueue handling
Hi,
Quoted 1 line · Rahul Singh
Add support for virtual cmdqueue handling for guests
This commit message is a bit light given the security implication of this approach. See some of the questions below.
[...]
xen/drivers/passthrough/arm/vsmmu-v3.c struct virt_smmu { · line 103
Quoted 11 lines
xen/drivers/passthrough/arm/vsmmu-v3.c +11
-0+static int arm_vsmmu_handle_cmds (struct virt_smmu *smmu) +{ +struct arm_vsmmu_queue *q = &smmu->cmdq; +struct domain *d = smmu->d; +uint64_t command[CMDQ_ENT_DWORDS ]; +paddr_t addr; + +if ( !smmu_get_cmdq_enabled (smmu->cr[0 ]) ) +return 0 ; + +while ( !queue_empty (q) )
What is the size of the queue and what would happen if the guest fill up the queue before triggering a call to arm_vsmmu_handle_cmds()?
xen/drivers/passthrough/arm/vsmmu-v3.c struct virt_smmu { · line 116
Quoted 13 lines
xen/drivers/passthrough/arm/vsmmu-v3.c +13
-0+ { +int ret; + + addr =Q_CONS_ENT (q); + ret =access_guest_memory_by_ipa (d, addr, command, +sizeof (command), false); +if ( ret ) +return ret; + +switch (smmu_cmd_get_command (command[0 ]) ) + { +case CMDQ_OP_CFGI_STE : +break ;
It is not clear to me why there is a break here when...
xen/drivers/passthrough/arm/vsmmu-v3.c struct virt_smmu { · line 122
Quoted 6 lines
xen/drivers/passthrough/arm/vsmmu-v3.c +6
-0+case CMDQ_OP_PREFETCH_CFG : +case CMDQ_OP_CFGI_CD : +case CMDQ_OP_CFGI_CD_ALL : +case CMDQ_OP_CFGI_ALL : +case CMDQ_OP_CMD_SYNC : +break ;
... the implementation for those is also empty.
xen/drivers/passthrough/arm/vsmmu-v3.c struct virt_smmu { · line 127
Quoted 5 lines
xen/drivers/passthrough/arm/vsmmu-v3.c +5
-0+case CMDQ_OP_TLBI_NH_ASID : +case CMDQ_OP_TLBI_NSNH_ALL : +case CMDQ_OP_TLBI_NH_VA : +if ( !iommu_iotlb_flush_all (smmu->d,1 ) ) +break ;
It is not clear to me why you are implementing TLB flush right now but not the other.
This call is also a massive hammer (the more if there are multiple IOMMUs involved) and I can see this been a way for a guest to abuse the vSMMUv3.
What is your end goal with the vSMMUv3, are you going to expose it to untrusted environment?
xen/drivers/passthrough/arm/vsmmu-v3.c struct virt_smmu { · line 130
Quoted 3 lines
xen/drivers/passthrough/arm/vsmmu-v3.c +3
-0+default : +gdprintk (XENLOG_ERR ,"vSMMUv3: unhandled command\n" ); +dump_smmu_command (command);
This would only be printed in a debug build. However, it is not clear to me how a guest would notice that Xen didn't handle the command.
So I think this should be a gprintk() to enable the user to figure out in production that something went wrong.
That said, isn't there a way to report error to the guest?
xen/drivers/passthrough/arm/vsmmu-v3.c struct virt_smmu { · line 138
Quoted 8 lines
xen/drivers/passthrough/arm/vsmmu-v3.c +8
-0+break ; + } + +if ( ret ) + { +gdprintk (XENLOG_ERR , +"vSMMUv3: command error %d while handling command\n" , + ret);
Same here. However, ret is so far always 0 until here. It would be preferable if this is introduced on the first user.
Quoted 11 lines
xen/drivers/passthrough/arm/vsmmu-v3.c +7
-0+dump_smmu_command (command); + } +queue_inc_cons (q); + } +return 0 ; +} +static int vsmmuv3_mmio_write (struct vcpu *v,mmio_info_t *info,register_t r,void *priv) {
xen/drivers/passthrough/arm/vsmmu-v3.c static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info, · line 199
Quoted 4 lines
xen/drivers/passthrough/arm/vsmmu-v3.c +1
-0@@ -103,9 +196,15 @@ static int vsmmuv3_mmio_write(struct vcpu *v, mmio_info_t *info,break ;case VREG32 (ARM_SMMU_CMDQ_PROD ): +spin_lock (&smmu->cmd_queue_lock);
The amount of work done with the spin_lock() taken looks quite large. This means a guest with N vCPUs could easily block N pCPUs for several seconds.
How do you plan to address this?
Cheers,
Signature · 1 line
Julien Grall
RFC PATCH 10/21 xen/arm: vsmmuv3: Add support for command CMD_CFGI_STE
In reply to Rahul SinghCMD_CFGI_STE is used to invalidate/validate the STE. Emulated vSMMUv3 driver in XEN will read the STE from the guest memory space and capture the Stage-1 configuration required to support nested translation.
--- xen/drivers/passthrough/arm/vsmmu-v3.c | 148 +++++++++++++++++++++++++ 1 file changed, 148 insertions(+)
xen/drivers/passthrough/arm/vsmmu-v3.c +148 -0
@@ -44,6 +44,21 @@ extern const struct viommu_desc __read_mostly *cur_viommu;/* Helper Macros */ #define smmu_get_cmdq_enabled (x) FIELD_GET(CR0_CMDQEN, x)#define smmu_cmd_get_command (x) FIELD_GET(CMDQ_0_OP, x) +#define smmu_cmd_get_sid (x) FIELD_GET(CMDQ_PREFETCH_0_SID, x) +#define smmu_get_ste_s1cdmax (x) FIELD_GET(STRTAB_STE_0_S1CDMAX, x) +#define smmu_get_ste_s1fmt (x) FIELD_GET(STRTAB_STE_0_S1FMT, x) +#define smmu_get_ste_s1stalld (x) FIELD_GET(STRTAB_STE_1_S1STALLD, x) +#define smmu_get_ste_s1ctxptr (x) FIELD_PREP(STRTAB_STE_0_S1CTXPTR_MASK, \ + FIELD_GET(STRTAB_STE_0_S1CTXPTR_MASK, x)) + +/* stage-1 translation configuration */ +struct arm_vsmmu_s1_trans_cfg { +paddr_t s1ctxptr; +uint8_t s1fmt; +uint8_t s1cdmax; +bool bypassed;/* translation is bypassed */ +bool aborted;/* translation is aborted */ +};/* virtual smmu queue */ struct arm_vsmmu_queue {
@@ -90,6 +105,138 @@ static void dump_smmu_command(uint64_t *command)gdprintk (XENLOG_ERR ,"cmd 0x%02llx: %016lx %016lx\n" ,smmu_cmd_get_command (command[0 ]), command[0 ], command[1 ]); } +static int arm_vsmmu_find_ste (struct virt_smmu *smmu,uint32_t sid, +uint64_t *ste) +{ +paddr_t addr, strtab_base; +struct domain *d = smmu->d; +uint32_t log2size; +int strtab_size_shift; +int ret; + + log2size =FIELD_GET (STRTAB_BASE_CFG_LOG2SIZE , smmu->strtab_base_cfg); + +if ( sid >= (1 <<MIN (log2size,SMMU_IDR1_SIDSIZE )) ) +return -EINVAL ; + +if ( smmu->features &STRTAB_BASE_CFG_FMT_2LVL ) + { +int idx, max_l2_ste, span; +paddr_t l1ptr, l2ptr; +uint64_t l1std; + + strtab_size_shift =MAX (5 , (int )log2size - smmu->sid_split -1 +3 ); + strtab_base = smmu->strtab_base &STRTAB_BASE_ADDR_MASK & + ~GENMASK_ULL (strtab_size_shift,0 ); + idx = (sid >>STRTAB_SPLIT ) *STRTAB_L1_DESC_DWORDS ; + l1ptr = (paddr_t)(strtab_base + idx *sizeof (l1std)); + + ret =access_guest_memory_by_ipa (d, l1ptr, &l1std, +sizeof (l1std), false); +if ( ret ) + { +gdprintk (XENLOG_ERR , +"Could not read L1PTR at 0X%" PRIx64"\n" , l1ptr); +return ret; + } + + span =FIELD_GET (STRTAB_L1_DESC_SPAN , l1std); +if ( !span ) + { +gdprintk (XENLOG_ERR ,"Bad StreamID span\n" ); +return -EINVAL ; + } + + max_l2_ste = (1 << span) -1 ; + l2ptr =FIELD_PREP (STRTAB_L1_DESC_L2PTR_MASK , +FIELD_GET (STRTAB_L1_DESC_L2PTR_MASK , l1std)); + idx = sid & ((1 << smmu->sid_split) -1 ); +if ( idx > max_l2_ste ) + { +gdprintk (XENLOG_ERR ,"idx=%d > max_l2_ste=%d\n" , + idx, max_l2_ste); +return -EINVAL ; + } + addr = l2ptr + idx *sizeof (*ste) *STRTAB_STE_DWORDS ; + } +else + { + strtab_size_shift = log2size +5 ; + strtab_base = smmu->strtab_base &STRTAB_BASE_ADDR_MASK & + ~GENMASK_ULL (strtab_size_shift,0 ); + addr = strtab_base + sid *sizeof (*ste) *STRTAB_STE_DWORDS ; + } + ret =access_guest_memory_by_ipa (d, addr, ste,sizeof (*ste), false); +if ( ret ) + { +gdprintk (XENLOG_ERR , +"Cannot fetch pte at address=0x%" PRIx64"\n" , addr); +return -EINVAL ; + } + +return 0 ; +} + +static int arm_vsmmu_decode_ste (struct virt_smmu *smmu,uint32_t sid, +struct arm_vsmmu_s1_trans_cfg *cfg, +uint64_t *ste) +{ +uint64_t val = ste[0 ]; + +if ( !(val &STRTAB_STE_0_V ) ) +return -EAGAIN ; + +switch (FIELD_GET (STRTAB_STE_0_CFG , val) ) + { +case STRTAB_STE_0_CFG_BYPASS : + cfg->bypassed = true; +return 0 ; +case STRTAB_STE_0_CFG_ABORT : + cfg->aborted = true; +return 0 ; +case STRTAB_STE_0_CFG_S1_TRANS : +break ; +case STRTAB_STE_0_CFG_S2_TRANS : +gdprintk (XENLOG_ERR ,"vSMMUv3 does not support stage 2 yet\n" ); + goto bad_ste; +default : +BUG ();/* STE corruption */ + } + + cfg->s1ctxptr =smmu_get_ste_s1ctxptr (val); + cfg->s1fmt =smmu_get_ste_s1fmt (val); + cfg->s1cdmax =smmu_get_ste_s1cdmax (val); +if ( cfg->s1cdmax !=0 ) + { +gdprintk (XENLOG_ERR , +"vSMMUv3 does not support multiple context descriptors\n" ); + goto bad_ste; + } + +return 0 ; + +bad_ste: +return -EINVAL ; +} + +static int arm_vsmmu_handle_cfgi_ste (struct virt_smmu *smmu,uint64_t *cmdptr) +{ +int ret; +uint64_t ste[STRTAB_STE_DWORDS ]; +struct arm_vsmmu_s1_trans_cfg s1_cfg = {0 }; +uint32_t sid =smmu_cmd_get_sid (cmdptr[0 ]); + + ret =arm_vsmmu_find_ste (smmu, sid, ste); +if ( ret ) +return ret; + + ret =arm_vsmmu_decode_ste (smmu, sid, &s1_cfg, ste); +if ( ret ) +return (ret == -EAGAIN ) ?0 : ret; + +return 0 ; +} +static int arm_vsmmu_handle_cmds (struct virt_smmu *smmu) {struct arm_vsmmu_queue *q = &smmu->cmdq;
@@ -113,6 +260,7 @@ static int arm_vsmmu_handle_cmds(struct virt_smmu *smmu)switch (smmu_cmd_get_command (command[0 ]) ) {case CMDQ_OP_CFGI_STE : + ret =arm_vsmmu_handle_cfgi_ste (smmu, command);break ;case CMDQ_OP_PREFETCH_CFG :case CMDQ_OP_CFGI_CD :
Signature · 1 line
2.25.1
Re: [RFC PATCH 10/21] xen/arm: vsmmuv3: Add support for command CMD_CFGI_STE
Hi Rahul,
Quoted 149 lines · Rahul Singh
CMD_CFGI_STEis used to invalidate/validate the STE. Emulated vSMMUv3 driver in XEN will read the STE from the guest memory space and capture the Stage-1 configuration required to support nested translation.Signed-off-by Rahul Singh <[email protected]>--- xen/drivers/passthrough/arm/vsmmu-v3.c | 148 +++++++++++++++++++++++++ 1 file changed, 148 insertions(+)xen/drivers/passthrough/arm/vsmmu-v3.c +123
-0@@ -44,6 +44,21 @@ extern const struct viommu_desc __read_mostly *cur_viommu;/* Helper Macros */ #define smmu_get_cmdq_enabled (x) FIELD_GET(CR0_CMDQEN, x)#define smmu_cmd_get_command (x) FIELD_GET(CMDQ_0_OP, x) +#define smmu_cmd_get_sid (x) FIELD_GET(CMDQ_PREFETCH_0_SID, x) +#define smmu_get_ste_s1cdmax (x) FIELD_GET(STRTAB_STE_0_S1CDMAX, x) +#define smmu_get_ste_s1fmt (x) FIELD_GET(STRTAB_STE_0_S1FMT, x) +#define smmu_get_ste_s1stalld (x) FIELD_GET(STRTAB_STE_1_S1STALLD, x) +#define smmu_get_ste_s1ctxptr (x) FIELD_PREP(STRTAB_STE_0_S1CTXPTR_MASK, \ + FIELD_GET(STRTAB_STE_0_S1CTXPTR_MASK, x)) + +/* stage-1 translation configuration */ +struct arm_vsmmu_s1_trans_cfg { +paddr_t s1ctxptr; +uint8_t s1fmt; +uint8_t s1cdmax; +bool bypassed;/* translation is bypassed */ +bool aborted;/* translation is aborted */ +};/* virtual smmu queue */ struct arm_vsmmu_queue {@@ -90,6 +105,138 @@ static void dump_smmu_command(uint64_t *command)gdprintk (XENLOG_ERR ,"cmd 0x%02llx: %016lx %016lx\n" ,smmu_cmd_get_command (command[0 ]), command[0 ], command[1 ]); } +static int arm_vsmmu_find_ste (struct virt_smmu *smmu,uint32_t sid, +uint64_t *ste) +{ +paddr_t addr, strtab_base; +struct domain *d = smmu->d; +uint32_t log2size; +int strtab_size_shift; +int ret; + + log2size =FIELD_GET (STRTAB_BASE_CFG_LOG2SIZE , smmu->strtab_base_cfg); + +if ( sid >= (1 <<MIN (log2size,SMMU_IDR1_SIDSIZE )) ) +return -EINVAL ; + +if ( smmu->features &STRTAB_BASE_CFG_FMT_2LVL ) + { +int idx, max_l2_ste, span; +paddr_t l1ptr, l2ptr; +uint64_t l1std; + + strtab_size_shift =MAX (5 , (int )log2size - smmu->sid_split -1 +3 ); + strtab_base = smmu->strtab_base &STRTAB_BASE_ADDR_MASK & + ~GENMASK_ULL (strtab_size_shift,0 ); + idx = (sid >>STRTAB_SPLIT ) *STRTAB_L1_DESC_DWORDS ; + l1ptr = (paddr_t)(strtab_base + idx *sizeof (l1std)); + + ret =access_guest_memory_by_ipa (d, l1ptr, &l1std, +sizeof (l1std), false); +if ( ret ) + { +gdprintk (XENLOG_ERR , +"Could not read L1PTR at 0X%" PRIx64"\n" , l1ptr); +return ret; + } + + span =FIELD_GET (STRTAB_L1_DESC_SPAN , l1std); +if ( !span ) + { +gdprintk (XENLOG_ERR ,"Bad StreamID span\n" ); +return -EINVAL ; + } + + max_l2_ste = (1 << span) -1 ; + l2ptr =FIELD_PREP (STRTAB_L1_DESC_L2PTR_MASK , +FIELD_GET (STRTAB_L1_DESC_L2PTR_MASK , l1std)); + idx = sid & ((1 << smmu->sid_split) -1 ); +if ( idx > max_l2_ste ) + { +gdprintk (XENLOG_ERR ,"idx=%d > max_l2_ste=%d\n" , + idx, max_l2_ste); +return -EINVAL ; + } + addr = l2ptr + idx *sizeof (*ste) *STRTAB_STE_DWORDS ; + } +else + { + strtab_size_shift = log2size +5 ; + strtab_base = smmu->strtab_base &STRTAB_BASE_ADDR_MASK & + ~GENMASK_ULL (strtab_size_shift,0 ); + addr = strtab_base + sid *sizeof (*ste) *STRTAB_STE_DWORDS ; + } + ret =access_guest_memory_by_ipa (d, addr, ste,sizeof (*ste), false); +if ( ret ) + { +gdprintk (XENLOG_ERR , +"Cannot fetch pte at address=0x%" PRIx64"\n" , addr); +return -EINVAL ; + } + +return 0 ; +} + +static int arm_vsmmu_decode_ste (struct virt_smmu *smmu,uint32_t sid, +struct arm_vsmmu_s1_trans_cfg *cfg, +uint64_t *ste) +{ +uint64_t val = ste[0 ]; + +if ( !(val &STRTAB_STE_0_V ) ) +return -EAGAIN ; + +switch (FIELD_GET (STRTAB_STE_0_CFG , val) ) + { +case STRTAB_STE_0_CFG_BYPASS : + cfg->bypassed = true; +return 0 ; +case STRTAB_STE_0_CFG_ABORT : + cfg->aborted = true; +return 0 ; +case STRTAB_STE_0_CFG_S1_TRANS : +break ; +case STRTAB_STE_0_CFG_S2_TRANS : +gdprintk (XENLOG_ERR ,"vSMMUv3 does not support stage 2 yet\n" ); + goto bad_ste; +default : +BUG ();/* STE corruption */ + } + + cfg->s1ctxptr =smmu_get_ste_s1ctxptr (val); + cfg->s1fmt =smmu_get_ste_s1fmt (val); + cfg->s1cdmax =smmu_get_ste_s1cdmax (val); +if ( cfg->s1cdmax !=0 ) + { +gdprintk (XENLOG_ERR , +"vSMMUv3 does not support multiple context descriptors\n" ); + goto bad_ste; + } +
xen/drivers/passthrough/arm/vsmmu-v3.c static void dump_smmu_command(uint64_t *command) · line 225
Quoted 10 lines
xen/drivers/passthrough/arm/vsmmu-v3.c +10
-0+return 0 ; + +bad_ste: +return -EINVAL ; +} + +static int arm_vsmmu_handle_cfgi_ste (struct virt_smmu *smmu,uint64_t *cmdptr) +{ +int ret; +uint64_t ste[STRTAB_STE_DWORDS ];
Do we really need to have an array here to represent the complete STE, even though we are only interested in the first DWORD containing S1ContextPtr? We do not care about other DWORDS and also in arm_vsmmu_find_ste, you access a single DWORD by doing:
ret = access_guest_memory_by_ipa(d, addr, ste, sizeof(*ste), false);
instead of:
ret = access_guest_memory_by_ipa(d, addr, ste, sizeof(*ste) * STRTAB_STE_DWORDS, false);
So to me, we could just do:uint64_t ste;
and we would be covered.
~Michal
RFC PATCH 11/21 xen/arm: vsmmuv3: Attach Stage-1 configuration to SMMUv3 hardware
In reply to Rahul SinghAttach the Stage-1 configuration to device STE to support nested translation for the guests.
--- xen/drivers/passthrough/arm/smmu-v3.c | 79 ++++++++++++++++++++++++++ xen/drivers/passthrough/arm/smmu-v3.h | 1 + xen/drivers/passthrough/arm/vsmmu-v3.c | 18 ++++++ xen/include/xen/iommu.h | 14 +++++ 4 files changed, 112 insertions(+)
xen/drivers/passthrough/arm/smmu-v3.c +79 -0
@@ -2754,6 +2754,37 @@ static struct arm_smmu_device *arm_smmu_get_by_dev(struct device *dev)return NULL ; } +static struct iommu_domain *arm_smmu_get_domain_by_sid (struct domain *d, +u32 sid) +{ +int i; +unsigned long flags; +struct iommu_domain *io_domain; +struct arm_smmu_domain *smmu_domain; +struct arm_smmu_master *master; +struct arm_smmu_xen_domain *xen_domain =dom_iommu (d)->arch.priv; + +/* +* Loop through the &xen_domain->contexts to locate a context +* assigned to this SMMU +*/ +list_for_each_entry (io_domain, &xen_domain->contexts, list) { + smmu_domain =to_smmu_domain (io_domain); + +spin_lock_irqsave (&smmu_domain->devices_lock, flags); +list_for_each_entry (master, &smmu_domain->devices, domain_head) { +for (i =0 ; i < master->num_streams; i++) { +if (sid != master->streams[i].id) +continue ; +spin_unlock_irqrestore (&smmu_domain->devices_lock, flags); +return io_domain; + } + } +spin_unlock_irqrestore (&smmu_domain->devices_lock, flags); + } +return NULL ; +} +static struct iommu_domain *arm_smmu_get_domain (struct domain *d,struct device *dev) {
@@ -2909,6 +2940,53 @@ static void arm_smmu_iommu_xen_domain_teardown(struct domain *d)xfree (xen_domain); } +static int arm_smmu_attach_guest_config (struct domain *d,u32 sid, +struct iommu_guest_config *cfg) +{ +int ret = -EINVAL ; +unsigned long flags; +struct arm_smmu_master *master; +struct arm_smmu_domain *smmu_domain; +struct arm_smmu_xen_domain *xen_domain =dom_iommu (d)->arch.priv; +struct iommu_domain *io_domain =arm_smmu_get_domain_by_sid (d, sid); + +if (!io_domain) +return -ENODEV ; + + smmu_domain =to_smmu_domain (io_domain); + +spin_lock (&xen_domain->lock); + +switch (cfg->config) { +case ARM_SMMU_DOMAIN_ABORT : + smmu_domain->abort = true; +break ; +case ARM_SMMU_DOMAIN_BYPASS : + smmu_domain->abort = false; +break ; +case ARM_SMMU_DOMAIN_NESTED : +/* Enable Nested stage translation. */ + smmu_domain->stage =ARM_SMMU_DOMAIN_NESTED ; + smmu_domain->s1_cfg.s1ctxptr = cfg->s1ctxptr; + smmu_domain->s1_cfg.s1fmt = cfg->s1fmt; + smmu_domain->s1_cfg.s1cdmax = cfg->s1cdmax; + smmu_domain->abort = false; +break ; +default : + goto out; + } + +spin_lock_irqsave (&smmu_domain->devices_lock, flags); +list_for_each_entry (master, &smmu_domain->devices, domain_head) +arm_smmu_install_ste_for_dev (master); +spin_unlock_irqrestore (&smmu_domain->devices_lock, flags); + + ret =0 ; +out: +spin_unlock (&xen_domain->lock); +return ret; +} +static const struct iommu_ops arm_smmu_iommu_ops = { .page_sizes =PAGE_SIZE_4K , .init = arm_smmu_iommu_xen_domain_init,
@@ -2921,6 +2999,7 @@ static const struct iommu_ops arm_smmu_iommu_ops = { .unmap_page = arm_iommu_unmap_page, .dt_xlate = arm_smmu_dt_xlate, .add_device = arm_smmu_add_device, + .attach_guest_config = arm_smmu_attach_guest_config }; static __init int arm_smmu_dt_init(struct dt_device_node *dev,
xen/drivers/passthrough/arm/smmu-v3.h +1 -0
@@ -393,6 +393,7 @@ enum arm_smmu_domain_stage {ARM_SMMU_DOMAIN_S2 ,ARM_SMMU_DOMAIN_NESTED ,ARM_SMMU_DOMAIN_BYPASS , + ARM_SMMU_DOMAIN_ABORT, };/* Xen specific code. */
xen/drivers/passthrough/arm/vsmmu-v3.c +18 -0
@@ -223,8 +223,11 @@ static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr) {int ret;uint64_t ste[STRTAB_STE_DWORDS ]; +struct domain *d = smmu->d; +struct domain_iommu *hd =dom_iommu (d);struct arm_vsmmu_s1_trans_cfg s1_cfg = {0 };uint32_t sid =smmu_cmd_get_sid (cmdptr[0 ]); +struct iommu_guest_config guest_cfg = {0 }; ret =arm_vsmmu_find_ste (smmu, sid, ste);if ( ret )
@@ -234,6 +237,21 @@ static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr)if ( ret )return (ret == -EAGAIN ) ?0 : ret; + guest_cfg.s1ctxptr = s1_cfg.s1ctxptr; + guest_cfg.s1fmt = s1_cfg.s1fmt; + guest_cfg.s1cdmax = s1_cfg.s1cdmax; + +if ( s1_cfg.bypassed ) + guest_cfg.config =ARM_SMMU_DOMAIN_BYPASS ; +else if ( s1_cfg.aborted ) + guest_cfg.config =ARM_SMMU_DOMAIN_ABORT ; +else + guest_cfg.config =ARM_SMMU_DOMAIN_NESTED ; + + ret = hd->platform_ops->attach_guest_config (d, sid, &guest_cfg); +if ( ret ) +return ret; +return 0 ; }
xen/include/xen/iommu.h +14 -0
@@ -230,6 +230,15 @@ int iommu_do_dt_domctl(struct xen_domctl *, struct domain *,#endif /* HAS_DEVICE_TREE */ +#ifdef CONFIG_ARM +struct iommu_guest_config { +paddr_t s1ctxptr; +uint8_t config; +uint8_t s1fmt; +uint8_t s1cdmax; +}; +#endif /* CONFIG_ARM */ +struct page_info ;/*
@@ -302,6 +311,11 @@ struct iommu_ops {*/ int (*dt_xlate)(device_t *dev,const struct dt_phandle_args *args);#endif + +#ifdef CONFIG_ARM +int (*attach_guest_config)(struct domain *d,u32 sid, +struct iommu_guest_config *cfg); +#endif };/*
Signature · 1 line
2.25.1
Re: [RFC PATCH 11/21] xen/arm: vsmmuv3: Attach Stage-1 configuration to SMMUv3 hardware
Quoted 4 lines · Rahul Singh
xen/include/xen/iommu.h +0
-0@@ -230,6 +230,15 @@ int iommu_do_dt_domctl(struct xen_domctl *, struct domain *,
xen/include/xen/iommu.h int iommu_do_dt_domctl(struct xen_domctl *, struct domain *, · line 240
Quoted 10 lines
xen/include/xen/iommu.h +8
-0#endif /* HAS_DEVICE_TREE */ +#ifdef CONFIG_ARM +struct iommu_guest_config { +paddr_t s1ctxptr; +uint8_t config; +uint8_t s1fmt; +uint8_t s1cdmax; +}; +#endif /* CONFIG_ARM */
Perhaps rather CONFIG_VIRTUAL_IOMMU?
Jan
Re: [RFC PATCH 11/21] xen/arm: vsmmuv3: Attach Stage-1 configuration to SMMUv3 hardware
In reply to Rahul SinghQuoted 108 lines · Rahul Singh
Attach the Stage-1 configuration to device STE to support nested translation for the guests.
Signed-off-by Rahul Singh <[email protected]>--- xen/drivers/passthrough/arm/smmu-v3.c | 79 ++++++++++++++++++++++++++ xen/drivers/passthrough/arm/smmu-v3.h | 1 + xen/drivers/passthrough/arm/vsmmu-v3.c | 18 ++++++ xen/include/xen/iommu.h | 14 +++++ 4 files changed, 112 insertions(+)xen/drivers/passthrough/arm/smmu-v3.c +78
-0@@ -2754,6 +2754,37 @@ static struct arm_smmu_device *arm_smmu_get_by_dev(struct device *dev)return NULL ; } +static struct iommu_domain *arm_smmu_get_domain_by_sid (struct domain *d, +u32 sid) +{ +int i; +unsigned long flags; +struct iommu_domain *io_domain; +struct arm_smmu_domain *smmu_domain; +struct arm_smmu_master *master; +struct arm_smmu_xen_domain *xen_domain =dom_iommu (d)->arch.priv; + +/* +* Loop through the &xen_domain->contexts to locate a context +* assigned to this SMMU +*/ +list_for_each_entry (io_domain, &xen_domain->contexts, list) { + smmu_domain =to_smmu_domain (io_domain); + +spin_lock_irqsave (&smmu_domain->devices_lock, flags); +list_for_each_entry (master, &smmu_domain->devices, domain_head) { +for (i =0 ; i < master->num_streams; i++) { +if (sid != master->streams[i].id) +continue ; +spin_unlock_irqrestore (&smmu_domain->devices_lock, flags); +return io_domain; + } + } +spin_unlock_irqrestore (&smmu_domain->devices_lock, flags); + } +return NULL ; +} +static struct iommu_domain *arm_smmu_get_domain (struct domain *d,struct device *dev) {@@ -2909,6 +2940,53 @@ static void arm_smmu_iommu_xen_domain_teardown(struct domain *d)xfree (xen_domain); } +static int arm_smmu_attach_guest_config (struct domain *d,u32 sid, +struct iommu_guest_config *cfg) +{ +int ret = -EINVAL ; +unsigned long flags; +struct arm_smmu_master *master; +struct arm_smmu_domain *smmu_domain; +struct arm_smmu_xen_domain *xen_domain =dom_iommu (d)->arch.priv; +struct iommu_domain *io_domain =arm_smmu_get_domain_by_sid (d, sid); + +if (!io_domain) +return -ENODEV ; + + smmu_domain =to_smmu_domain (io_domain); + +spin_lock (&xen_domain->lock); + +switch (cfg->config) { +case ARM_SMMU_DOMAIN_ABORT : + smmu_domain->abort = true; +break ; +case ARM_SMMU_DOMAIN_BYPASS : + smmu_domain->abort = false; +break ; +case ARM_SMMU_DOMAIN_NESTED : +/* Enable Nested stage translation. */ + smmu_domain->stage =ARM_SMMU_DOMAIN_NESTED ; + smmu_domain->s1_cfg.s1ctxptr = cfg->s1ctxptr; + smmu_domain->s1_cfg.s1fmt = cfg->s1fmt; + smmu_domain->s1_cfg.s1cdmax = cfg->s1cdmax; + smmu_domain->abort = false; +break ; +default : + goto out; + } + +spin_lock_irqsave (&smmu_domain->devices_lock, flags); +list_for_each_entry (master, &smmu_domain->devices, domain_head) +arm_smmu_install_ste_for_dev (master); +spin_unlock_irqrestore (&smmu_domain->devices_lock, flags); + + ret =0 ; +out: +spin_unlock (&xen_domain->lock); +return ret; +} +static const struct iommu_ops arm_smmu_iommu_ops = { .page_sizes =PAGE_SIZE_4K , .init = arm_smmu_iommu_xen_domain_init,
xen/drivers/passthrough/arm/smmu-v3.c static const struct iommu_ops arm_smmu_iommu_ops = { · line 3002
Quoted 4 lines
xen/drivers/passthrough/arm/smmu-v3.c +1
-0@@ -2921,6 +2999,7 @@ static const struct iommu_ops arm_smmu_iommu_ops = { .unmap_page = arm_iommu_unmap_page, .dt_xlate = arm_smmu_dt_xlate, .add_device = arm_smmu_add_device, + .attach_guest_config = arm_smmu_attach_guest_config
Please append a trailing comma here, even if it is the last element
Quoted 89 lines
xen/drivers/passthrough/arm/smmu-v3.c +0
-0}; static __init int arm_smmu_dt_init(struct dt_device_node *dev,xen/drivers/passthrough/arm/smmu-v3.h +1
-0@@ -393,6 +393,7 @@ enum arm_smmu_domain_stage {ARM_SMMU_DOMAIN_S2 ,ARM_SMMU_DOMAIN_NESTED ,ARM_SMMU_DOMAIN_BYPASS , + ARM_SMMU_DOMAIN_ABORT, };/* Xen specific code. */ xen/drivers/passthrough/arm/vsmmu-v3.c +18
-0@@ -223,8 +223,11 @@ static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr) {int ret;uint64_t ste[STRTAB_STE_DWORDS ]; +struct domain *d = smmu->d; +struct domain_iommu *hd =dom_iommu (d);struct arm_vsmmu_s1_trans_cfg s1_cfg = {0 };uint32_t sid =smmu_cmd_get_sid (cmdptr[0 ]); +struct iommu_guest_config guest_cfg = {0 }; ret =arm_vsmmu_find_ste (smmu, sid, ste);if ( ret )@@ -234,6 +237,21 @@ static int arm_vsmmu_handle_cfgi_ste(struct virt_smmu *smmu, uint64_t *cmdptr)if ( ret )return (ret == -EAGAIN ) ?0 : ret; + guest_cfg.s1ctxptr = s1_cfg.s1ctxptr; + guest_cfg.s1fmt = s1_cfg.s1fmt; + guest_cfg.s1cdmax = s1_cfg.s1cdmax; + +if ( s1_cfg.bypassed ) + guest_cfg.config =ARM_SMMU_DOMAIN_BYPASS ; +else if ( s1_cfg.aborted ) + guest_cfg.config =ARM_SMMU_DOMAIN_ABORT ; +else + guest_cfg.config =ARM_SMMU_DOMAIN_NESTED ; + + ret = hd->platform_ops->attach_guest_config (d, sid, &guest_cfg); +if ( ret ) +return ret; +return 0 ; }xen/include/xen/iommu.h +14
-1@@ -230,6 +230,15 @@ int iommu_do_dt_domctl(struct xen_domctl *, struct domain *,#endif /* HAS_DEVICE_TREE */ +#ifdef CONFIG_ARM +struct iommu_guest_config { +paddr_t s1ctxptr; +uint8_t config; +uint8_t s1fmt; +uint8_t s1cdmax; +}; +#endif /* CONFIG_ARM */ +struct page_info ;/* @@ -302,6 +311,11 @@ struct iommu_ops {*/ int (*dt_xlate)(device_t *dev,const struct dt_phandle_args *args);#endif + +#ifdef CONFIG_ARM +int (*attach_guest_config)(struct domain *d,u32 sid, +struct iommu_guest_config *cfg); +#endif };/* -- 2.25.1
RFC PATCH 12/21 xen/arm: vsmmuv3: Add support for event queue and global error
In reply to Rahul SinghEvent queue is used to send the events to guest when there is an events/ faults. Add support for event queue to send events to guest.
Global error in SMMUv3 hw will be updated in smmu_gerror and smmu_gerrorn register. Add support for global error registers to send global error to guest.
--- xen/drivers/passthrough/arm/smmu-v3.h | 20 +++ xen/drivers/passthrough/arm/vsmmu-v3.c | 163 ++++++++++++++++++++++++- xen/include/public/arch-arm.h | 5 +- 3 files changed, 183 insertions(+), 5 deletions(-)
xen/drivers/passthrough/arm/smmu-v3.h +20 -0
@@ -348,6 +348,26 @@#define EVTQ_0_ID GENMASK_ULL(7, 0) +#define EVT_ID_BAD_STREAMID 0x02 +#define EVT_ID_BAD_STE 0x04 +#define EVT_ID_TRANSLATION_FAULT 0x10 +#define EVT_ID_ADDR_SIZE_FAULT 0x11 +#define EVT_ID_ACCESS_FAULT 0x12 +#define EVT_ID_PERMISSION_FAULT 0x13 + +#define EVTQ_0_SSV (1UL << 11) +#define EVTQ_0_SSID GENMASK_ULL(31, 12) +#define EVTQ_0_SID GENMASK_ULL(63, 32) +#define EVTQ_1_STAG GENMASK_ULL(15, 0) +#define EVTQ_1_STALL (1UL << 31) +#define EVTQ_1_PnU (1UL << 33) +#define EVTQ_1_InD (1UL << 34) +#define EVTQ_1_RnW (1UL << 35) +#define EVTQ_1_S2 (1UL << 39) +#define EVTQ_1_CLASS GENMASK_ULL(41, 40) +#define EVTQ_1_TT_READ (1UL << 44) +#define EVTQ_2_ADDR GENMASK_ULL(63, 0) +#define EVTQ_3_IPA GENMASK_ULL(51, 12)/* PRI queue */ #define PRIQ_ENT_SZ_SHIFT 4#define PRIQ_ENT_DWORDS ((1 << PRIQ_ENT_SZ_SHIFT) >> 3)
xen/drivers/passthrough/arm/vsmmu-v3.c +160 -3
251 lines not on this page · raw message
xen/include/public/arch-arm.h +3 -2
@@ -488,9 +488,10 @@ typedef uint64_t xen_callback_t;#define GUEST_EVTCHN_PPI 31#define GUEST_VPL011_SPI 32 +#define GUEST_VSMMU_SPI 33 -#define GUEST_VIRTIO_MMIO_SPI_FIRST 33 -#define GUEST_VIRTIO_MMIO_SPI_LAST 43 +#define GUEST_VIRTIO_MMIO_SPI_FIRST 34 +#define GUEST_VIRTIO_MMIO_SPI_LAST 44/* PSCI functions */ #define PSCI_cpu_suspend 0
Signature · 1 line
2.25.1
RFC PATCH 13/21 xen/arm: vsmmuv3: Add "iommus" property node for dom0 devices
In reply to Rahul Singh"iommus" property will be added for dom0 devices to virtual IOMMU node to enable the dom0 linux kernel to configure the IOMMU
--- xen/arch/arm/domain_build.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
xen/arch/arm/domain_build.c +5 -2
@@ -1172,9 +1172,12 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,continue ; } -if ( iommu_node ) +/* +* Expose IOMMU specific properties to hwdom when vIOMMU is +* enabled. +*/ +if ( iommu_node && !is_viommu_enabled () ) { -/* Don't expose IOMMU specific properties to hwdom */ if (dt_property_name_is_equal (prop,"iommus" ) )continue ;
Signature · 1 line
2.25.1
RFC PATCH 14/21 xen/arm: vIOMMU: IOMMU device tree node for dom0
In reply to Rahul SinghXEN will create an IOMMU device tree node in the device tree to enable the dom0 to discover the virtual SMMUv3 during dom0 boot. IOMMU device tree node will only be created when cmdline option viommu is enabled.
--- xen/arch/arm/domain_build.c | 94 +++++++++++++++++++++++++++++++ xen/arch/arm/include/asm/viommu.h | 1 + 2 files changed, 95 insertions(+)
xen/arch/arm/domain_build.c +94 -0
@@ -2233,6 +2233,95 @@ int __init make_chosen_node(const struct kernel_info *kinfo)return res; } +#ifdef CONFIG_VIRTUAL_IOMMU +static int make_hwdom_viommu_node (const struct kernel_info *kinfo) +{ +uint32_t len; +int res; +char buf[24 ]; +void *fdt = kinfo->fdt; +const void *prop =NULL ; +const struct dt_device_node *iommu =NULL ; +struct host_iommu *iommu_data; +gic_interrupt_t intr; + +if (list_empty (&host_iommu_list) ) +return 0 ; + +list_for_each_entry ( iommu_data, &host_iommu_list, entry ) + { +if ( iommu_data->hwdom_node_created ) +return 0 ; + + iommu = iommu_data->dt_node; + +snprintf (buf,sizeof (buf),"iommu@%" PRIx64, iommu_data->addr); + + res =fdt_begin_node (fdt, buf); +if ( res ) +return res; + + prop =dt_get_property (iommu,"compatible" , &len); +if ( !prop ) + { + res = -FDT_ERR_XEN (ENOENT ); +return res; + } + + res =fdt_property (fdt,"compatible" , prop, len); +if ( res ) +return res; + +if ( iommu->phandle ) + { + res =fdt_property_cell (fdt,"phandle" , iommu->phandle); +if ( res ) +return res; + } + +/* Use the same reg regions as the IOMMU node in host DTB. */ + prop =dt_get_property (iommu,"reg" , &len); +if ( !prop ) + { +printk (XENLOG_ERR "vIOMMU: Can't find IOMMU reg property.\n" ); + res = -FDT_ERR_XEN (ENOENT ); +return res; + } + + res =fdt_property (fdt,"reg" , prop, len); +if ( res ) +return res; + + prop =dt_get_property (iommu,"#iommu-cells" , &len); +if ( !prop ) + { + res = -FDT_ERR_XEN (ENOENT ); +return res; + } + + res =fdt_property (fdt,"#iommu-cells" , prop, len); +if ( res ) +return res; + + res =fdt_property_string (fdt,"interrupt-names" ,"combined" ); +if ( res ) +return res; + +set_interrupt (intr, iommu_data->irq,0xf ,DT_IRQ_TYPE_LEVEL_HIGH ); + + res =fdt_property_interrupts (kinfo, &intr,1 ); +if ( res ) +return res; + + iommu_data->hwdom_node_created = true; + +fdt_end_node (fdt); + } + +return res; +} +#endif +int __init map_irq_to_domain (struct domain *d,unsigned int irq,bool need_mapping,const char *devname) {
@@ -2587,6 +2676,11 @@ static int __init handle_node(struct domain *d, struct kernel_info *kinfo,if (dt_match_node (timer_matches, node) )return make_timer_node (kinfo); +#ifdef CONFIG_VIRTUAL_IOMMU +if (device_get_class (node) ==DEVICE_IOMMU &&is_viommu_enabled () ) +return make_hwdom_viommu_node (kinfo); +#endif +/* Skip nodes used by Xen */ if (dt_device_used_by (node) ==DOMID_XEN ) {
xen/arch/arm/include/asm/viommu.h +1 -0
@@ -19,6 +19,7 @@ struct host_iommu {paddr_t addr;paddr_t size;uint32_t irq; +bool hwdom_node_created; };struct viommu_ops {
Signature · 1 line
2.25.1
Re: [RFC PATCH 14/21] xen/arm: vIOMMU: IOMMU device tree node for dom0
Quoted 16 lines · Rahul Singh
XEN will create an IOMMU device tree node in the device tree to enable the dom0 to discover the virtual SMMUv3 during dom0 boot. IOMMU device tree node will only be created when cmdline option viommu is enabled.
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/domain_build.c | 94 +++++++++++++++++++++++++++++++ xen/arch/arm/include/asm/viommu.h | 1 + 2 files changed, 95 insertions(+)xen/arch/arm/domain_build.c +0
-0
xen/arch/arm/domain_build.c int __init make_chosen_node(const struct kernel_info *kinfo) · line 2237
Quoted 5 lines
xen/arch/arm/domain_build.c +2
-0@@ -2233,6 +2233,95 @@ int __init make_chosen_node(const struct kernel_info *kinfo)return res; } +#ifdef CONFIG_VIRTUAL_IOMMU +static int make_hwdom_viommu_node (const struct kernel_info *kinfo)
This should have the __init attribute
RFC PATCH 15/21 xen/arm: vsmmuv3: Emulated SMMUv3 device tree node for dom0less
In reply to Rahul SinghXEN will create an Emulated SMMUv3 device tree node in the device tree to enable the dom0less domains to discover the virtual SMMUv3 during boot. Emulated SMMUv3 device tree node will only be created when cmdline option vsmmuv3 is enabled.
--- xen/arch/arm/domain_build.c | 52 +++++++++++++++++++++++++++ xen/include/public/device_tree_defs.h | 1 + 2 files changed, 53 insertions(+)
xen/arch/arm/domain_build.c +52 -0
@@ -2322,6 +2322,49 @@ static int make_hwdom_viommu_node(const struct kernel_info *kinfo) }#endif +#ifdef CONFIG_VIRTUAL_ARM_SMMU_V3 +static int __init make_vsmmuv3_node (const struct kernel_info *kinfo) +{ +int res; +char buf[24 ]; +__be32 reg[GUEST_ROOT_ADDRESS_CELLS +GUEST_ROOT_SIZE_CELLS ]; +__be32 *cells; +void *fdt = kinfo->fdt; + +snprintf (buf,sizeof (buf),"iommu@%llx" ,GUEST_VSMMUV3_BASE ); + + res =fdt_begin_node (fdt, buf); +if ( res ) +return res; + + res =fdt_property_string (fdt,"compatible" ,"arm,smmu-v3" ); +if ( res ) +return res; + +/* Create reg property */ + cells = ®[0 ]; +dt_child_set_range (&cells,GUEST_ROOT_ADDRESS_CELLS ,GUEST_ROOT_SIZE_CELLS , +GUEST_VSMMUV3_BASE ,GUEST_VSMMUV3_SIZE ); + res =fdt_property (fdt,"reg" , reg, + (GUEST_ROOT_ADDRESS_CELLS + +GUEST_ROOT_SIZE_CELLS ) *sizeof (*reg)); +if ( res ) +return res; + + res =fdt_property_cell (fdt,"phandle" ,GUEST_PHANDLE_VSMMUV3 ); +if ( res ) +return res; + + res =fdt_property_cell (fdt,"#iommu-cells" ,1 ); +if ( res ) +return res; + + res =fdt_end_node (fdt); + +return res; +} +#endif +int __init map_irq_to_domain (struct domain *d,unsigned int irq,bool need_mapping,const char *devname) {
@@ -3395,6 +3438,15 @@ static int __init prepare_dtb_domU(struct domain *d, struct kernel_info *kinfo) goto err; } +#ifdef CONFIG_VIRTUAL_ARM_SMMU_V3 +if (is_viommu_enabled () ) + { + ret =make_vsmmuv3_node (kinfo); +if ( ret ) + goto err; + } +#endif + ret =fdt_end_node (kinfo->fdt);if ( ret <0 ) goto err;
xen/include/public/device_tree_defs.h +1 -0
@@ -14,6 +14,7 @@*/ #define GUEST_PHANDLE_GIC (65000)#define GUEST_PHANDLE_IOMMU (GUEST_PHANDLE_GIC + 1) +#define GUEST_PHANDLE_VSMMUV3 (GUEST_PHANDLE_IOMMU + 1)#define GUEST_ROOT_ADDRESS_CELLS 2#define GUEST_ROOT_SIZE_CELLS 2
Signature · 1 line
2.25.1
Re: [RFC PATCH 15/21] xen/arm: vsmmuv3: Emulated SMMUv3 device tree node for dom0less
Hi Rahul,
Quoted 6 lines · Rahul Singh
XEN will create an Emulated SMMUv3 device tree node in the device tree to enable the dom0less domains to discover the virtual SMMUv3 during boot. Emulated SMMUv3 device tree node will only be created when cmdline option vsmmuv3 is enabled.
I think you meant viommu option.
In any case, I think that you want to introduce a device-tree property for dom0less domU e.g. string property viommu = "smmuv3" to control whether to add a virtual IOMMU node or not for a given domU. You did exactly the same for xl created domUs so you should do the same for dom0less domUs. Otherwise the behavior will be inconsistent and we will lack per-domain control over virtual IOMMU.
~Michal
RFC PATCH 16/21 arm/libxl: vsmmuv3: Emulated SMMUv3 device tree node in libxl
In reply to Rahul Singhlibxl will create an Emulated SMMUv3 device tree node in the device tree to enable the guest OS to discover the virtual SMMUv3 during guest boot.
Emulated SMMUv3 device tree node will only be created when "viommu=smmuv3" is set in xl domain configuration.
--- tools/libs/light/libxl_arm.c | 39 ++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
tools/libs/light/libxl_arm.c +39 -0
@@ -831,6 +831,36 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,return 0 ; } +static int make_vsmmuv3_node (libxl__gc *gc,void *fdt, +const struct arch_info *ainfo, +struct xc_dom_image *dom) +{ +int res; +const char *name =GCSPRINTF ("iommu@%llx" ,GUEST_VSMMUV3_BASE ); + + res =fdt_begin_node (fdt, name); +if (res)return res; + + res =fdt_property_compat (gc, fdt,1 ,"arm,smmu-v3" ); +if (res)return res; + + res =fdt_property_regs (gc, fdt,GUEST_ROOT_ADDRESS_CELLS , +GUEST_ROOT_SIZE_CELLS ,1 ,GUEST_VSMMUV3_BASE , +GUEST_VSMMUV3_SIZE ); +if (res)return res; + + res =fdt_property_cell (fdt,"phandle" ,GUEST_PHANDLE_VSMMUV3 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#iommu-cells" ,1 ); +if (res)return res; + + res =fdt_end_node (fdt); +if (res)return res; + +return 0 ; +} +static int make_vpci_node (libxl__gc *gc,void *fdt,const struct arch_info *ainfo,struct xc_dom_image *dom)
@@ -872,6 +902,12 @@ static int make_vpci_node(libxl__gc *gc, void *fdt, GUEST_VPCI_PREFETCH_MEM_SIZE);if (res)return res; +if (res)return res; + + res =fdt_property_values (gc, fdt,"iommu-map" ,4 ,0 , +GUEST_PHANDLE_VSMMUV3 ,0 ,0x10000 ); +if (res)return res; + res =fdt_end_node (fdt);if (res)return res;
@@ -1251,6 +1287,9 @@ next_resize:if (d_config->num_pcidevs)FDT (make_vpci_node (gc, fdt, ainfo, dom) ); +if (info->arch_arm.viommu_type ==LIBXL_VIOMMU_TYPE_SMMUV3 ) +FDT (make_vsmmuv3_node (gc, fdt, ainfo, dom) ); + iommu_created = false;for (i =0 ; i < d_config->num_disks; i++) {libxl_device_disk *disk = &d_config->disks[i];
Signature · 1 line
2.25.1
RFC PATCH 17/21 xen/arm: vsmmuv3: Alloc virq for virtual SMMUv3
In reply to Rahul SinghAlloc and reserve virq for event queue and global error to send event to guests. Also Modify the libxl to accomadate the new define virq.
--- tools/libs/light/libxl_arm.c | 24 ++++++++++++++++++++++-- xen/arch/arm/domain_build.c | 11 +++++++++++ xen/drivers/passthrough/arm/vsmmu-v3.c | 10 ++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-)
tools/libs/light/libxl_arm.c +22 -2
@@ -66,8 +66,8 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc, {uint32_t nr_spis =0 ;unsigned int i; -uint32_t vuart_irq, virtio_irq =0 ; -bool vuart_enabled = false, virtio_enabled = false; +uint32_t vuart_irq, virtio_irq =0 , vsmmu_irq =0 ; +bool vuart_enabled = false, virtio_enabled = false, vsmmu_enabled = false;uint64_t virtio_mmio_base =GUEST_VIRTIO_MMIO_BASE ;uint32_t virtio_mmio_irq =GUEST_VIRTIO_MMIO_SPI_FIRST ;
@@ -81,6 +81,12 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc, vuart_enabled = true; } +if (d_config->num_pcidevs || d_config->b_info.device_tree) { + nr_spis += (GUEST_VSMMU_SPI -32 ) +1 ; + vsmmu_irq =GUEST_VSMMU_SPI ; + vsmmu_enabled = true; + } +for (i =0 ; i < d_config->num_disks; i++) {libxl_device_disk *disk = &d_config->disks[i];
@@ -136,6 +142,11 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,return ERROR_FAIL ; } +if (vsmmu_enabled && irq == vsmmu_irq) { +LOG (ERROR ,"Physical IRQ %u conflicting with vSMMUv3 SPI\n" , irq); +return ERROR_FAIL ; + } +if (irq <32 )continue ;
@@ -837,6 +848,7 @@ static int make_vsmmuv3_node(libxl__gc *gc, void *fdt, {int res;const char *name =GCSPRINTF ("iommu@%llx" ,GUEST_VSMMUV3_BASE ); +gic_interrupt intr; res =fdt_begin_node (fdt, name);if (res)return res;
@@ -855,6 +867,14 @@ static int make_vsmmuv3_node(libxl__gc *gc, void *fdt, res =fdt_property_cell (fdt,"#iommu-cells" ,1 );if (res)return res; + res =fdt_property_string (fdt,"interrupt-names" ,"combined" ); +if (res)return res; + +set_interrupt (intr,GUEST_VSMMU_SPI ,0xf ,DT_IRQ_TYPE_LEVEL_HIGH ); + + res =fdt_property_interrupts (gc, fdt, &intr,1 ); +if (res)return res; + res =fdt_end_node (fdt);if (res)return res;
xen/arch/arm/domain_build.c +11 -0
@@ -2329,6 +2329,7 @@ static int __init make_vsmmuv3_node(const struct kernel_info *kinfo)char buf[24 ];__be32 reg[GUEST_ROOT_ADDRESS_CELLS +GUEST_ROOT_SIZE_CELLS ];__be32 *cells; +gic_interrupt_t intr;void *fdt = kinfo->fdt;snprintf (buf,sizeof (buf),"iommu@%llx" ,GUEST_VSMMUV3_BASE );
@@ -2359,6 +2360,16 @@ static int __init make_vsmmuv3_node(const struct kernel_info *kinfo)if ( res )return res; + res =fdt_property_string (fdt,"interrupt-names" ,"combined" ); +if ( res ) +return res; + +set_interrupt (intr,GUEST_VSMMU_SPI ,0xf ,DT_IRQ_TYPE_LEVEL_HIGH ); + + res =fdt_property_interrupts (kinfo, &intr,1 ); +if ( res ) +return res; + res =fdt_end_node (fdt);return res;
xen/drivers/passthrough/arm/vsmmu-v3.c +10 -0
@@ -728,6 +728,7 @@ static const struct mmio_handler_ops vsmmuv3_mmio_handler = {static int vsmmuv3_init_single (struct domain *d,paddr_t addr,paddr_t size,uint32_t virq) { +int ret;struct virt_smmu *smmu; smmu = xzalloc(struct virt_smmu);
@@ -743,12 +744,21 @@ static int vsmmuv3_init_single(struct domain *d, paddr_t addr,spin_lock_init (&smmu->cmd_queue_lock); + ret =vgic_reserve_virq (d, virq); +if ( !ret ) + goto out; +register_mmio_handler (d, &vsmmuv3_mmio_handler, addr, size, smmu);/* Register the vIOMMU to be able to clean it up later. */ list_add_tail (&smmu->viommu_list, &d->arch.viommu_list);return 0 ; + +out: +xfree (smmu); +vgic_free_virq (d, virq); +return ret; }int domain_vsmmuv3_init (struct domain *d)
Signature · 1 line
2.25.1
RFC PATCH 18/21 xen/arm: iommu: skip the iommu-map property for PCI devices
In reply to Rahul SinghCurrent code skip the IOMMUS specific properties for the non PCI devices when handling the dom0 node but there is no support to skip the IOMMUS specific properties for the PCI devices.
This patch will add the support to skip the IOMMUS specific properties for the PCI devices.
--- xen/arch/arm/domain_build.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
xen/arch/arm/domain_build.c +12 -3
@@ -1112,9 +1112,18 @@ static int __init write_properties(struct domain *d, struct kernel_info *kinfo,* Use "iommu_node" as an indicator of the master device which properties * should be skipped. */ - iommu_node =dt_parse_phandle (node,"iommus" ,0 ); -if ( iommu_node &&device_get_class (iommu_node) !=DEVICE_IOMMU ) - iommu_node =NULL ; +if (dt_device_type_is_equal (node,"pci" ) ) + { + iommu_node =dt_parse_phandle (node,"iommu-map" ,1 ); +if ( iommu_node &&device_get_class (iommu_node) !=DEVICE_IOMMU ) + iommu_node =NULL ; + } +else + { + iommu_node =dt_parse_phandle (node,"iommus" ,0 ); +if ( iommu_node &&device_get_class (iommu_node) !=DEVICE_IOMMU ) + iommu_node =NULL ; + } dt_for_each_property_node (node, prop) {
Signature · 1 line
2.25.1
RFC PATCH 19/21 xen/arm: vsmmuv3: Add support to send stage-1 event to guest
In reply to Rahul SinghStage-1 translation is handled by guest, therefore stage-1 fault has to be forwarded to guest.
--- xen/drivers/passthrough/arm/smmu-v3.c | 48 ++++++++++++++++++++++++-- xen/drivers/passthrough/arm/vsmmu-v3.c | 45 ++++++++++++++++++++++++ xen/drivers/passthrough/arm/vsmmu-v3.h | 12 +++++++ 3 files changed, 103 insertions(+), 2 deletions(-)
xen/drivers/passthrough/arm/smmu-v3.c +46 -2
@@ -871,7 +871,6 @@ static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid)return 0 ; } -__maybe_unused static struct arm_smmu_master *arm_smmu_find_master (struct arm_smmu_device *smmu,u32 sid) {
@@ -892,10 +891,51 @@ arm_smmu_find_master(struct arm_smmu_device *smmu, u32 sid)return NULL ; } +static int arm_smmu_handle_evt (struct arm_smmu_device *smmu,u64 *evt) +{ +int ret; +struct arm_smmu_master *master; +u32 sid =FIELD_GET (EVTQ_0_SID , evt[0 ]); + +switch (FIELD_GET (EVTQ_0_ID , evt[0 ])) { +case EVT_ID_TRANSLATION_FAULT : +break ; +case EVT_ID_ADDR_SIZE_FAULT : +break ; +case EVT_ID_ACCESS_FAULT : +break ; +case EVT_ID_PERMISSION_FAULT : +break ; +default : +return -EOPNOTSUPP ; + } + +/* Stage-2 event */ +if (evt[1 ] &EVTQ_1_S2 ) +return -EFAULT ; + +mutex_lock (&smmu->streams_mutex); + master =arm_smmu_find_master (smmu, sid); +if (!master) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =arm_vsmmu_handle_evt (master->domain->d, smmu->dev, evt); +if (ret) { + ret = -EINVAL ; + goto out_unlock; + } + +out_unlock: +mutex_unlock (&smmu->streams_mutex); +return ret; +} +/* IRQ and event handlers */ static void arm_smmu_evtq_tasklet (void *dev) { -int i; +int i, ret;struct arm_smmu_device *smmu = dev;struct arm_smmu_queue *q = &smmu->evtq.q;struct arm_smmu_ll_queue *llq = &q->llq;
@@ -905,6 +945,10 @@ static void arm_smmu_evtq_tasklet(void *dev)while (!queue_remove_raw (q, evt)) {u8 id =FIELD_GET (EVTQ_0_ID , evt[0 ]); + ret =arm_smmu_handle_evt (smmu, evt); +if (!ret) +continue ; +dev_info (smmu->dev,"event 0x%02x received:\n" , id);for (i =0 ; i <ARRAY_SIZE (evt); ++i) dev_info(smmu->dev, "\t0x%016llx\n",
xen/drivers/passthrough/arm/vsmmu-v3.c +45 -0
63 lines not on this page · raw message
xen/drivers/passthrough/arm/vsmmu-v3.h +12 -0
@@ -8,6 +8,12 @@void vsmmuv3_set_type (void ); +static inline int arm_vsmmu_handle_evt (struct domain *d, +struct device *dev,uint64_t *evt) +{ +return -EINVAL ; +} +#else static inline void vsmmuv3_set_type (void )
@@ -15,6 +21,12 @@ static inline void vsmmuv3_set_type(void)return ; } +static inline int arm_vsmmu_handle_evt (struct domain *d, +struct device *dev,uint64_t *evt) +{ +return -EINVAL ; +} +#endif /* CONFIG_VIRTUAL_ARM_SMMU_V3 */ #endif /* __ARCH_ARM_VSMMU_V3_H__ */
Signature · 1 line
2.25.1
RFC PATCH 20/21 libxl/arm: vIOMMU: Modify the partial device tree for iommus
In reply to Rahul SinghTo configure IOMMU in guest for passthrough devices, user will need to copy the unmodified "iommus" property from host device tree to partial device tree. To enable the dom0 linux kernel to confiure the IOMMU correctly replace the phandle in partial device tree with virtual IOMMU phandle when "iommus" property is set.
--- tools/libs/light/libxl_arm.c | 47 +++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-)
tools/libs/light/libxl_arm.c +46 -1
@@ -1167,6 +1167,41 @@ static int copy_partial_fdt(libxl__gc *gc, void *fdt, void *pfdt)return 0 ; } +static int modify_partial_fdt (libxl__gc *gc,void *pfdt) +{ +int nodeoff, proplen, i, r; +const fdt32_t *prop; +fdt32_t *prop_c; + + nodeoff =fdt_path_offset (pfdt,"/passthrough" ); +if (nodeoff <0 ) +return nodeoff; + +for (nodeoff =fdt_first_subnode (pfdt, nodeoff); + nodeoff >=0 ; + nodeoff =fdt_next_subnode (pfdt, nodeoff)) { + + prop =fdt_getprop (pfdt, nodeoff,"iommus" , &proplen); +if (!prop) +continue ; + + prop_c =libxl__zalloc (gc, proplen); + +for (i =0 ; i < proplen /8 ; ++i) { + prop_c[i *2 ] =cpu_to_fdt32 (GUEST_PHANDLE_VSMMUV3 ); + prop_c[i *2 +1 ] = prop[i *2 +1 ]; + } + + r =fdt_setprop (pfdt, nodeoff,"iommus" , prop_c, proplen); +if (r) { +LOG (ERROR ,"Can't set the iommus property in partial FDT" ); +return r; + } + } + +return 0 ; +} +#else static int check_partial_fdt (libxl__gc *gc,void *fdt,size_t size)
@@ -1185,6 +1220,13 @@ static int copy_partial_fdt(libxl__gc *gc, void *fdt, void *pfdt) return -FDT_ERR_INTERNAL; } +static int modify_partial_fdt (libxl__gc *gc,void *pfdt) +{ +LOG (ERROR ,"partial device tree not supported" ); + +return ERROR_FAIL ; +} +#endif /* ENABLE_PARTIAL_DEVICE_TREE */ #define FDT_MAX_SIZE (1<<20)
@@ -1307,8 +1349,11 @@ next_resize:if (d_config->num_pcidevs)FDT (make_vpci_node (gc, fdt, ainfo, dom) ); -if (info->arch_arm.viommu_type ==LIBXL_VIOMMU_TYPE_SMMUV3 ) +if (info->arch_arm.viommu_type ==LIBXL_VIOMMU_TYPE_SMMUV3 ) {FDT (make_vsmmuv3_node (gc, fdt, ainfo, dom) ); +if (pfdt) +FDT (modify_partial_fdt (gc, pfdt) ); + } iommu_created = false;for (i =0 ; i < d_config->num_disks; i++) {
Signature · 1 line
2.25.1
RFC PATCH 21/21 xen/arm: vIOMMU: Modify the partial device tree for dom0less
In reply to Rahul SinghTo configure IOMMU in guest for passthrough devices, user will need to copy the unmodified "iommus" property from host device tree to partial device tree. To enable the dom0 linux kernel to confiure the IOMMU correctly replace the phandle in partial device tree with virtual IOMMU phandle when "iommus" property is set.
--- xen/arch/arm/domain_build.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
xen/arch/arm/domain_build.c +30 -1
@@ -3235,7 +3235,35 @@ static int __init handle_prop_pfdt(struct kernel_info *kinfo,return ( propoff != -FDT_ERR_NOTFOUND ) ? propoff :0 ; } -static int __init scan_pfdt_node (struct kernel_info *kinfo,const void *pfdt, +static void modify_pfdt_node (void *pfdt,int nodeoff) +{ +int proplen, i, rc; +const fdt32_t *prop; +fdt32_t *prop_c; + + prop =fdt_getprop (pfdt, nodeoff,"iommus" , &proplen); +if ( !prop ) +return ; + + prop_c =xzalloc_bytes (proplen); + +for ( i =0 ; i < proplen /8 ; ++i ) + { + prop_c[i *2 ] =cpu_to_fdt32 (GUEST_PHANDLE_VSMMUV3 ); + prop_c[i *2 +1 ] = prop[i *2 +1 ]; + } + + rc =fdt_setprop (pfdt, nodeoff,"iommus" , prop_c, proplen); +if ( rc ) + { +dprintk (XENLOG_ERR ,"Can't set the iommus property in partial FDT" ); +return ; + } + +return ; +} + +static int __init scan_pfdt_node (struct kernel_info *kinfo,void *pfdt,int nodeoff,uint32_t address_cells,uint32_t size_cells,bool scan_passthrough_prop)
@@ -3261,6 +3289,7 @@ static int __init scan_pfdt_node(struct kernel_info *kinfo, const void *pfdt, node_next =fdt_first_subnode (pfdt, nodeoff);while ( node_next >0 ) { +modify_pfdt_node (pfdt, node_next);scan_pfdt_node (kinfo, pfdt, node_next, address_cells, size_cells, scan_passthrough_prop); node_next =fdt_next_subnode (pfdt, node_next);
Signature · 1 line
2.25.1
Re: [RFC PATCH 21/21] xen/arm: vIOMMU: Modify the partial device tree for dom0less
Hi Rahul,
Quoted 55 lines · Rahul Singh
To configure IOMMU in guest for passthrough devices, user will need to copy the unmodified "iommus" property from host device tree to partial device tree. To enable the dom0 linux kernel to confiure the IOMMU correctly replace the phandle in partial device tree with virtual IOMMU phandle when "iommus" property is set.
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/domain_build.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)xen/arch/arm/domain_build.c +29
-1@@ -3235,7 +3235,35 @@ static int __init handle_prop_pfdt(struct kernel_info *kinfo,return ( propoff != -FDT_ERR_NOTFOUND ) ? propoff :0 ; } -static int __init scan_pfdt_node (struct kernel_info *kinfo,const void *pfdt, +static void modify_pfdt_node (void *pfdt,int nodeoff) +{ +int proplen, i, rc; +const fdt32_t *prop; +fdt32_t *prop_c; + + prop =fdt_getprop (pfdt, nodeoff,"iommus" , &proplen); +if ( !prop ) +return ; + + prop_c =xzalloc_bytes (proplen); + +for ( i =0 ; i < proplen /8 ; ++i ) + { + prop_c[i *2 ] =cpu_to_fdt32 (GUEST_PHANDLE_VSMMUV3 ); + prop_c[i *2 +1 ] = prop[i *2 +1 ]; + } + + rc =fdt_setprop (pfdt, nodeoff,"iommus" , prop_c, proplen); +if ( rc ) + { +dprintk (XENLOG_ERR ,"Can't set the iommus property in partial FDT" ); +return ; + } + +return ; +} + +static int __init scan_pfdt_node (struct kernel_info *kinfo,void *pfdt,int nodeoff,uint32_t address_cells,uint32_t size_cells,bool scan_passthrough_prop)
xen/arch/arm/domain_build.c static int __init scan_pfdt_node(struct kernel_info *kinfo, const void *pfdt, · line 3292
Quoted 4 lines
xen/arch/arm/domain_build.c +1
-0@@ -3261,6 +3289,7 @@ static int __init scan_pfdt_node(struct kernel_info *kinfo, const void *pfdt, node_next =fdt_first_subnode (pfdt, nodeoff);while ( node_next >0 ) { +modify_pfdt_node (pfdt, node_next);
You do not protect this call in any way and there is no check inside it whether viommu is enabled or not. This means that even with viommu disabled, if a user copies the iommus property to partial dtb (which is understandable as the flow is to copy almost everything), you will perform the phandle replacement. I do not think we should do that (no need for an extra code).
Quoted 7 lines
xen/arch/arm/domain_build.c +0
-1scan_pfdt_node (kinfo, pfdt, node_next, address_cells, size_cells, scan_passthrough_prop); node_next =fdt_next_subnode (pfdt, node_next); --2.25.1
~Michal
Re: [RFC PATCH 21/21] xen/arm: vIOMMU: Modify the partial device tree for dom0less
In reply to Rahul SinghQuoted 16 lines · Rahul Singh
To configure IOMMU in guest for passthrough devices, user will need to copy the unmodified "iommus" property from host device tree to partial device tree. To enable the dom0 linux kernel to confiure the IOMMU correctly replace the phandle in partial device tree with virtual IOMMU phandle when "iommus" property is set.
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/domain_build.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)xen/arch/arm/domain_build.c +0
-0
xen/arch/arm/domain_build.c static int __init handle_prop_pfdt(struct kernel_info *kinfo, · line 3238
Quoted 5 lines
xen/arch/arm/domain_build.c +1
-1@@ -3235,7 +3235,35 @@ static int __init handle_prop_pfdt(struct kernel_info *kinfo,return ( propoff != -FDT_ERR_NOTFOUND ) ? propoff :0 ; } -static int __init scan_pfdt_node(struct kernel_info *kinfo, const void *pfdt, +static void modify_pfdt_node (void *pfdt,int nodeoff)
This should have the __init attribute
Hi Rahul,
Quoted 45 lines · Rahul Singh
The SMMUv3 supports two stages of translation. Each stage of translation can be independently enabled. An incoming address is logically translated from VA to IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to the output PA.
Stage 1 is intended to be used by a software entity to provide isolation or translation to buffers within the entity, for example DMA isolation within an OS. Stage 2 is intended to be available in systems supporting the Virtualization Extensions and is intended to virtualize device DMA to guest VM address spaces. When both stage 1 and stage 2 are enabled, the translation configuration is called nested.
Stage 1 translation support is required to provide isolation between different devices within OS. XEN already supports Stage 2 translation but there is no support for Stage 1 translation. The goal of this work is to support Stage 1 translation for XEN guests. Stage 1 has to be configured within the guest to provide isolation.
We cannot trust the guest OS to control the SMMUv3 hardware directly as compromised guest OS can corrupt the SMMUv3 configuration and make the system vulnerable. The guest gets the ownership of the stage 1 page tables and also owns stage 1 configuration structures. The XEN handles the root configuration structure (for security reasons), including the stage 2 configuration.
XEN will emulate the SMMUv3 hardware and exposes the virtual SMMUv3 to the guest. Guest can use the native SMMUv3 driver to configure the stage 1 translation. When the guest configures the SMMUv3 for Stage 1, XEN will trap the access and configure hardware.
SMMUv3 Driver(Guest OS) -> Configure the Stage-1 translation -> XEN trap access -> XEN SMMUv3 driver configure the HW.
SMMUv3 driver has to be updated to support the Stage-1 translation support based on work done by the KVM team to support Nested Stage translation:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feauger%2Flinux%2Fcommits%2Fv5.11-stallv12-2stage-v14&data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PdK4%2Bsps3%2FdXYJUDv3iCy%2Byaqbh1bOVb1AFzTtx1nts%3D&reserved=0
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F852299%2F&data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5Kp7023HiA4Qbfi28wcPL20JyC2xLwwiyEUZcxTSCOA%3D&reserved=0As the stage 1 translation is configured by XEN on behalf of the guest, translation faults encountered during the translation process need to be propagated up to the guest and re-injected into the guest. When the guest invalidates stage 1 related caches, invalidations must be forwarded to the SMMUv3 hardware.
Quoted 5 lines
This patch series is sent as RFC to get the initial feedback from the community. This patch series consists of 21 patches which is a big number for the reviewer to review the patches but to understand the feature end-to-end we thought of sending this as a big series. Once we will get initial feedback, we will divide the series into a small number of patches for review.
Due to the very limited availability of the board we have, that is equipped with DMA platform devices and SMMUv3 (I know that you tested PCI use case thoroughly), I managed for now to do the testing on dom0 only.
By commenting out the code in Linux responsible for setting up Xen SWIOTLB DMA ops, I was able to successfully verify the nested SMMU working properly for DMA platform devices on the example of using ZDMA. Both the upstream dmatest client app as well as the VFIO user space driver that I wrote for ZDMA passed the test!
I added some logs to verify the sync up between Linux and Xen during a VFIO test:
LINUX: SMMUv3: Setting the STE S1 Config 0x1405c000 for SID=0x210
XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
XEN: SMMUv3: Setting the STE S1 Config 0x1405c000 for SID=0x210
Before transfer example:
src value: 0xdb71faf
dst value: 0
Waiting for transfer completion...
After transfer example:
src value: 0xdb71faf
dst value: 0xdb71faf
TEST RESULT: PASS
LINUX: SMMUv3: Setting the STE S1 Config 0x12502000 for SID=0x210
XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
XEN: SMMUv3: Setting the STE S1 Config 0x12502000 for SID=0x210
~Michal
Hi Rahul,
Quoted 78 lines · Michal Orzel
Hi Rahul,
Quoted 45 lines · Rahul Singh
The SMMUv3 supports two stages of translation. Each stage of translation can be independently enabled. An incoming address is logically translated from VA to IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to the output PA.
Stage 1 is intended to be used by a software entity to provide isolation or translation to buffers within the entity, for example DMA isolation within an OS. Stage 2 is intended to be available in systems supporting the Virtualization Extensions and is intended to virtualize device DMA to guest VM address spaces. When both stage 1 and stage 2 are enabled, the translation configuration is called nested.
Stage 1 translation support is required to provide isolation between different devices within OS. XEN already supports Stage 2 translation but there is no support for Stage 1 translation. The goal of this work is to support Stage 1 translation for XEN guests. Stage 1 has to be configured within the guest to provide isolation.
We cannot trust the guest OS to control the SMMUv3 hardware directly as compromised guest OS can corrupt the SMMUv3 configuration and make the system vulnerable. The guest gets the ownership of the stage 1 page tables and also owns stage 1 configuration structures. The XEN handles the root configuration structure (for security reasons), including the stage 2 configuration.
XEN will emulate the SMMUv3 hardware and exposes the virtual SMMUv3 to the guest. Guest can use the native SMMUv3 driver to configure the stage 1 translation. When the guest configures the SMMUv3 for Stage 1, XEN will trap the access and configure hardware.
SMMUv3 Driver(Guest OS) -> Configure the Stage-1 translation -> XEN trap access -> XEN SMMUv3 driver configure the HW.
SMMUv3 driver has to be updated to support the Stage-1 translation support based on work done by the KVM team to support Nested Stage translation:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feauger%2Flinux%2Fcommits%2Fv5.11-stallv12-2stage-v14&data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PdK4%2Bsps3%2FdXYJUDv3iCy%2Byaqbh1bOVb1AFzTtx1nts%3D&reserved=0
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F852299%2F&data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5Kp7023HiA4Qbfi28wcPL20JyC2xLwwiyEUZcxTSCOA%3D&reserved=0As the stage 1 translation is configured by XEN on behalf of the guest, translation faults encountered during the translation process need to be propagated up to the guest and re-injected into the guest. When the guest invalidates stage 1 related caches, invalidations must be forwarded to the SMMUv3 hardware.
Quoted 5 lines
This patch series is sent as RFC to get the initial feedback from the community. This patch series consists of 21 patches which is a big number for the reviewer to review the patches but to understand the feature end-to-end we thought of sending this as a big series. Once we will get initial feedback, we will divide the series into a small number of patches for review.
Due to the very limited availability of the board we have, that is equipped with DMA platform devices and SMMUv3 (I know that you tested PCI use case thoroughly), I managed for now to do the testing on dom0 only.
By commenting out the code in Linux responsible for setting up Xen SWIOTLB DMA ops, I was able to successfully verify the nested SMMU working properly for DMA platform devices on the example of using ZDMA. Both the upstream dmatest client app as well as the VFIO user space driver that I wrote for ZDMA passed the test!
I added some logs to verify the sync up between Linux and Xen during a VFIO test:
LINUX: SMMUv3: Setting the STE S1 Config
0x1405c000for SID=0x210
XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
XEN: SMMUv3: Setting the STE S1 Config0x1405c000for SID=0x210Before transfer example:
src value:0xdb71faf
dst value: 0
Waiting for transfer completion...
After transfer example:
src value:0xdb71faf
dst value:0xdb71faf
TEST RESULT: PASS
Quoted 3 lines
LINUX: SMMUv3: Setting the STE S1 Config
0x12502000for SID=0x210
XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
XEN: SMMUv3: Setting the STE S1 Config0x12502000for SID=0x210
I finished testing this series by also covering dom0less and xl domUs. Tests passed so good job!
I do not have access to any board with more than one IOMMU so I could not validate this behavior.
~Michal
Hi Michal,
Quoted 83 lines · Michal Orzel
Hi Rahul,
Quoted 78 lines · Michal Orzel
Hi Rahul,
Quoted 45 lines · Rahul Singh
The SMMUv3 supports two stages of translation. Each stage of translation can be independently enabled. An incoming address is logically translated from VA to IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to the output PA.
Stage 1 is intended to be used by a software entity to provide isolation or translation to buffers within the entity, for example DMA isolation within an OS. Stage 2 is intended to be available in systems supporting the Virtualization Extensions and is intended to virtualize device DMA to guest VM address spaces. When both stage 1 and stage 2 are enabled, the translation configuration is called nested.
Stage 1 translation support is required to provide isolation between different devices within OS. XEN already supports Stage 2 translation but there is no support for Stage 1 translation. The goal of this work is to support Stage 1 translation for XEN guests. Stage 1 has to be configured within the guest to provide isolation.
We cannot trust the guest OS to control the SMMUv3 hardware directly as compromised guest OS can corrupt the SMMUv3 configuration and make the system vulnerable. The guest gets the ownership of the stage 1 page tables and also owns stage 1 configuration structures. The XEN handles the root configuration structure (for security reasons), including the stage 2 configuration.
XEN will emulate the SMMUv3 hardware and exposes the virtual SMMUv3 to the guest. Guest can use the native SMMUv3 driver to configure the stage 1 translation. When the guest configures the SMMUv3 for Stage 1, XEN will trap the access and configure hardware.
SMMUv3 Driver(Guest OS) -> Configure the Stage-1 translation -> XEN trap access -> XEN SMMUv3 driver configure the HW.
SMMUv3 driver has to be updated to support the Stage-1 translation support based on work done by the KVM team to support Nested Stage translation:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Feauger%2Flinux%2Fcommits%2Fv5.11-stallv12-2stage-v14&data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PdK4%2Bsps3%2FdXYJUDv3iCy%2Byaqbh1bOVb1AFzTtx1nts%3D&reserved=0
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F852299%2F&data=05%7C01%7Cmichal.orzel%40amd.com%7Cecb9075a29974c8f5ad608dad3b5916f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638055074068482160%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5Kp7023HiA4Qbfi28wcPL20JyC2xLwwiyEUZcxTSCOA%3D&reserved=0As the stage 1 translation is configured by XEN on behalf of the guest, translation faults encountered during the translation process need to be propagated up to the guest and re-injected into the guest. When the guest invalidates stage 1 related caches, invalidations must be forwarded to the SMMUv3 hardware.
Quoted 5 lines
This patch series is sent as RFC to get the initial feedback from the community. This patch series consists of 21 patches which is a big number for the reviewer to review the patches but to understand the feature end-to-end we thought of sending this as a big series. Once we will get initial feedback, we will divide the series into a small number of patches for review.
Due to the very limited availability of the board we have, that is equipped with DMA platform devices and SMMUv3 (I know that you tested PCI use case thoroughly), I managed for now to do the testing on dom0 only.
By commenting out the code in Linux responsible for setting up Xen SWIOTLB DMA ops, I was able to successfully verify the nested SMMU working properly for DMA platform devices on the example of using ZDMA. Both the upstream dmatest client app as well as the VFIO user space driver that I wrote for ZDMA passed the test!
I added some logs to verify the sync up between Linux and Xen during a VFIO test:
LINUX: SMMUv3: Setting the STE S1 Config
0x1405c000for SID=0x210
XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
XEN: SMMUv3: Setting the STE S1 Config0x1405c000for SID=0x210Before transfer example:
src value:0xdb71faf
dst value: 0
Waiting for transfer completion...
After transfer example:
src value:0xdb71faf
dst value:0xdb71faf
TEST RESULT: PASS
Quoted 9 lines
Quoted 3 lines
LINUX: SMMUv3: Setting the STE S1 Config
0x12502000for SID=0x210
XEN: vSMMUv3: guest config=ARM_SMMU_DOMAIN_NESTED
XEN: SMMUv3: Setting the STE S1 Config0x12502000for SID=0x210I finished testing this series by also covering dom0less and xl domUs. Tests passed so good job!
I do not have access to any board with more than one IOMMU so I could not validate this behavior.
Thank you much appreciated for testing the series as it helps a lot to confirm that this feature is working as expected.
Regards,
Rahul
Hi Rahul,
Quoted 5 lines · Rahul Singh
This patch series is sent as RFC to get the initial feedback from the community. This patch series consists of 21 patches which is a big number for the reviewer to review the patches but to understand the feature end-to-end we thought of sending this as a big series. Once we will get initial feedback, we will divide the series into a small number of patches for review.
From the cover letter, it is not clear to me what sort of input you are expecting for the RFC. Is this about the design itself?
If so, I think it would be more helpful to write an high level document on how you plan to emulate the vIOMMU in Xen. So there is one place to read/agree/verify rather than trying to collate all the information from the 20+ patches.
Briefly skimming through I think the main things that need to be addressed in order of priority:
- How to secure the vIOMMU
- 1 vs multiple vIOMMU
The questions are very similar to the vITS because the SMMUv3 is based on a queue. And given you are selling this feature as a security one, I don't think we can go forward with the review without any understanding/agreement on what needs to be implemented in order to have a safe/secure vIOMMU.
Cheers,
Signature · 1 line
Julien Grall
Quoted 23 lines · Julien Grall
Quoted 8 lines · Rahul Singh
This patch series is sent as RFC to get the initial feedback from the community. This patch series consists of 21 patches which is a big number for
the reviewer to review the patches but to understand the feature end-to-end we
thought of sending this as a big series. Once we will get initial feedback, we
will divide the series into a small number of patches for review.From the cover letter, it is not clear to me what sort of input you are expecting for the RFC. Is this about the design itself?
If so, I think it would be more helpful to write an high level document on how you plan to emulate the vIOMMU in Xen. So there is one place to read/agree/verify rather than trying to collate all the information from the 20+ patches.
Briefly skimming through I think the main things that need to be addressed in order of priority:
- How to secure the vIOMMU
- 1 vs multiple vIOMMU
Quoted 4 lines
The questions are very similar to the vITS because the SMMUv3 is based on a queue. And given you are selling this feature as a security one, I don't think we can go forward with the review without any understanding/agreement on what needs to be implemented in order to have a safe/secure vIOMMU.
I think we are all aligned here, but let me try to clarify further.
As the vIOMMU is exposed to the guest, and exposing a queue-based interface to the guest is not simple, it would be good to clarify in a document the following points:
- how is the queue exposed to the guest
- how are guest-inputs sanitized
- how do the virtual queue resources map to the physical queue resources
- lifecycle of the resource mappings
- any memory allocations triggered by guest actions and their lifecycle
It is difficult to extrapole these details from 21 patches. Having these key detailed written down in the 0/21 email would greatly help with the review. It would make the review go a lot faster.
Hi Stefano,Julien
Quoted 44 lines · Stefano Stabellini
Quoted 23 lines · Julien Grall
Quoted 8 lines · Rahul Singh
This patch series is sent as RFC to get the initial feedback from the community. This patch series consists of 21 patches which is a big number for
the reviewer to review the patches but to understand the feature end-to-end we
thought of sending this as a big series. Once we will get initial feedback, we
will divide the series into a small number of patches for review.From the cover letter, it is not clear to me what sort of input you are expecting for the RFC. Is this about the design itself?
If so, I think it would be more helpful to write an high level document on how you plan to emulate the vIOMMU in Xen. So there is one place to read/agree/verify rather than trying to collate all the information from the 20+ patches.
Briefly skimming through I think the main things that need to be addressed in order of priority:
- How to secure the vIOMMU
- 1 vs multiple vIOMMUQuoted 4 lines
The questions are very similar to the vITS because the SMMUv3 is based on a queue. And given you are selling this feature as a security one, I don't think we can go forward with the review without any understanding/agreement on what needs to be implemented in order to have a safe/secure vIOMMU.
I think we are all aligned here, but let me try to clarify further.
As the vIOMMU is exposed to the guest, and exposing a queue-based interface to the guest is not simple, it would be good to clarify in a document the following points:
- how is the queue exposed to the guest
- how are guest-inputs sanitized
- how do the virtual queue resources map to the physical queue resources
- lifecycle of the resource mappings
- any memory allocations triggered by guest actions and their lifecycle
Quoted 3 lines
It is difficult to extrapole these details from 21 patches. Having these key detailed written down in the 0/21 email would greatly help with the review. It would make the review go a lot faster.
Ack. I will send the design docs by next week that will include all the requested information.
Regards.
Rahul