use stashed domain create flags...
v5 cover use stashed domain create flags...
...and add per-domain IOMMU control
This is a combination of my previously separate series [1] and [2].
[1] https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02253.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html
Paul Durrant (10):
x86/hvm/domain: remove the 'hap_enabled' flag x86/domain: remove the 'oos_off' flag
domain: remove the 'is_xenstore' flag
x86/domain: remove the 's3_integrity' flag
domain: introduce XEN_DOMCTL_CDF_iommu flag
use is_iommu_enabled() where appropriate...
remove late (on-demand) construction of IOMMU page tables make passthrough/pci.c:deassign_device() static
iommu: tidy up iommu_use_hap_pt() and need_iommu_pt_sync() macros introduce a 'passthrough' configuration option to xl.cfg...
docs/man/xl.cfg.5.pod.in | 52 +++++++ tools/libxl/libxl.h | 5 + tools/libxl/libxl_create.c | 22 +++ tools/libxl/libxl_types.idl | 7 + tools/xl/xl_parse.c | 38 +++++ xen/arch/arm/domain.c | 11 +- xen/arch/arm/p2m.c | 4 +- xen/arch/arm/setup.c | 3 + xen/arch/x86/dom0_build.c | 4 +- xen/arch/x86/domain.c | 24 +-- xen/arch/x86/domctl.c | 4 +- xen/arch/x86/hvm/hvm.c | 6 +- xen/arch/x86/hvm/mtrr.c | 5 +- xen/arch/x86/hvm/vioapic.c | 2 +- xen/arch/x86/hvm/vmx/vmcs.c | 2 +- xen/arch/x86/hvm/vmx/vmx.c | 2 +- xen/arch/x86/mm/mem_sharing.c | 2 +- xen/arch/x86/mm/p2m-ept.c | 4 +- xen/arch/x86/mm/paging.c | 6 +- xen/arch/x86/mm/shadow/common.c | 7 +- xen/arch/x86/mm/shadow/none.c | 2 +- xen/arch/x86/setup.c | 5 +- xen/arch/x86/tboot.c | 2 +- xen/arch/x86/x86_64/mm.c | 2 +- xen/common/domain.c | 40 ++++- xen/common/domctl.c | 2 +- xen/common/memory.c | 4 +- xen/common/vm_event.c | 2 +- xen/drivers/passthrough/amd/iommu_guest.c | 2 +- xen/drivers/passthrough/device_tree.c | 18 +-- xen/drivers/passthrough/io.c | 8 +- xen/drivers/passthrough/iommu.c | 174 +++++++--------------- xen/drivers/passthrough/pci.c | 119 +++++++-------- xen/drivers/passthrough/vtd/iommu.c | 12 +- xen/drivers/passthrough/vtd/x86/hvm.c | 2 +- xen/drivers/passthrough/x86/iommu.c | 97 +----------- xen/include/asm-arm/iommu.h | 3 - xen/include/asm-x86/domain.h | 3 - xen/include/asm-x86/hvm/domain.h | 7 - xen/include/asm-x86/iommu.h | 4 - xen/include/asm-x86/paging.h | 2 +- xen/include/asm-x86/shadow.h | 2 +- xen/include/public/domctl.h | 10 +- xen/include/xen/iommu.h | 41 ++--- xen/include/xen/sched.h | 25 ++-- xen/include/xsm/dummy.h | 2 +- xen/xsm/flask/hooks.c | 18 +-- 47 files changed, 397 insertions(+), 421 deletions(-)
---
Cc: Andrew Cooper <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Brian Woods <[email protected]>
Cc: Daniel De Graaf <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Jun Nakajima <[email protected]>
Cc: Kevin Tian <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Petre Pircalabu <[email protected]>
Cc: Razvan Cojocaru <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Suravee Suthikulpanit <[email protected]>
Cc: Tamas K Lengyel <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Wei Liu <[email protected]>
Signature · 7 lines
2.20.1.2.gb21ebb671 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 1/10 x86/hvm/domain: remove the 'hap_enabled' flag
The hap_enabled() macro can determine whether the feature is available using the domain 'options'; there is no need for a separate flag.
NOTE: Furthermore, by extending sanitiziing of the domain 'options', the
macro can be transformed into an inline function and re-located to xen/sched.h. This also makes hap_enabled() common, thus allowing removal of an ugly ifdef CONFIG_X86 from the common iommu code.
---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: George Dunlap <[email protected]>
v4:
- Add missing perentheses and move hap_enabled()
v3:
- Re-worked as suggested by Jan
- Not adding Roger's R-b as the patch has changed substantially
v2:
- Defer changes to shadow_domain_init() to patch #4
--- xen/arch/x86/domain.c | 13 +++++++------ xen/arch/x86/mm/paging.c | 4 ++-- xen/common/domain.c | 7 +++++++ xen/drivers/passthrough/iommu.c | 2 -- xen/include/asm-x86/hvm/domain.h | 7 ------- xen/include/asm-x86/paging.h | 2 +- xen/include/xen/sched.h | 5 +++++ 7 files changed, 22 insertions(+), 18 deletions(-)
xen/arch/x86/domain.c +7 -6
@@ -460,6 +460,12 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +if ( (config->flags & XEN_DOMCTL_CDF_hap) && !hvm_hap_supported () ) + { +dprintk (XENLOG_INFO ,"HAP enabled but not supported\n" ); +return -EINVAL ; + } +return 0 ; }
@@ -564,12 +570,7 @@ int arch_domain_create(struct domain *d,HYPERVISOR_COMPAT_VIRT_START (d) =is_pv_domain (d) ? __HYPERVISOR_COMPAT_VIRT_START : ~0u ; -/* Need to determine if HAP is enabled before initialising paging */ -if (is_hvm_domain (d) ) - d->arch.hvm.hap_enabled = -hvm_hap_supported () && (config->flags & XEN_DOMCTL_CDF_hap); - -if ( (rc =paging_domain_init (d, config->flags)) !=0 ) +if ( (rc =paging_domain_init (d)) !=0 ) goto fail; paging_initialised = true;
xen/arch/x86/mm/paging.c +2 -2
@@ -632,7 +632,7 @@ void paging_log_dirty_init(struct domain *d, const struct log_dirty_ops *ops)/* CODE FOR PAGING SUPPORT */ /************************************************/ /* Domain paging struct initialization. */ -int paging_domain_init (struct domain *d,unsigned int domcr_flags) +int paging_domain_init (struct domain *d) {int rc;
@@ -653,7 +653,7 @@ int paging_domain_init(struct domain *d, unsigned int domcr_flags)if (hap_enabled (d) )hap_domain_init (d);else - rc =shadow_domain_init (d, domcr_flags); + rc =shadow_domain_init (d, d->options);return rc; }
xen/common/domain.c +7 -0
@@ -313,6 +313,13 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +if ( !(config->flags & XEN_DOMCTL_CDF_hvm_guest) && + (config->flags & XEN_DOMCTL_CDF_hap) ) + { +dprintk (XENLOG_INFO ,"HAP enabled for non-HVM guest\n" ); +return -EINVAL ; + } +return arch_sanitise_domain_config (config); }
xen/drivers/passthrough/iommu.c +0 -2
@@ -573,9 +573,7 @@ int iommu_do_domctl(void iommu_share_p2m_table (struct domain * d) { -#ifdef CONFIG_X86 ASSERT (hap_enabled (d)); -#endif /* * iommu_use_hap_pt(d) cannot be used here because during domain * construction need_iommu(d) will always return false here.
xen/include/asm-x86/hvm/domain.h +0 -7
@@ -156,7 +156,6 @@ struct hvm_domain {struct viridian_domain *viridian; -bool_t hap_enabled;bool_t mem_sharing_enabled;bool_t qemu_mapcache_invalidate;bool_t is_s3_suspended;
@@ -195,12 +194,6 @@ struct hvm_domain { }; }; -#ifdef CONFIG_HVM -#define hap_enabled (d) (is_hvm_domain(d) && (d)->arch.hvm.hap_enabled) -#else -#define hap_enabled (d) ({(void)(d); false;}) -#endif -#endif /* __ASM_X86_HVM_DOMAIN_H__ */ /*
xen/include/asm-x86/paging.h +1 -1
@@ -207,7 +207,7 @@ void paging_vcpu_init(struct vcpu *v);/* Set up the paging-assistance-specific parts of a domain struct at * start of day. Called for every domain from arch_domain_create() */ -int paging_domain_init (struct domain *d,unsigned int domcr_flags); +int paging_domain_init (struct domain *d);/* Handler for paging-control ops: operations from user-space to enable * and disable ephemeral shadow modes (test mode and log-dirty mode) and
xen/include/xen/sched.h +5 -0
@@ -954,6 +954,11 @@ static inline bool is_hvm_vcpu(const struct vcpu *v)return is_hvm_domain (v->domain); } +static inline bool hap_enabled (const struct domain *d) +{ +return evaluate_nospec (d->options & XEN_DOMCTL_CDF_hap); +} +static inline bool is_hwdom_pinned_vcpu (const struct vcpu *v) { return (is_hardware_domain(v->domain) &&
Signature · 7 lines
2.20.1.2.gb21ebb671 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 2/10 x86/domain: remove the 'oos_off' flag
In reply to Paul DurrantThe flag is not needed since the domain 'options' can now be tested directly.
---
Cc: Tim Deegan <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
v3:
- Force 'oos_off' to be set for PV guests (to avoid call to is_hvm_domain() except in ASSERT)
- Dropped Tim's A-b because of the change
v2:
- Move some of the hunks from patch #3
- Also update the definition of shadow_domain_init() in none.c
--- xen/arch/x86/mm/paging.c | 2 +- xen/arch/x86/mm/shadow/common.c | 7 ++++--- xen/arch/x86/mm/shadow/none.c | 2 +- xen/common/domain.c | 16 ++++++++++++---- xen/include/asm-x86/domain.h | 1 - xen/include/asm-x86/shadow.h | 2 +- 6 files changed, 19 insertions(+), 11 deletions(-)
xen/arch/x86/mm/paging.c +1 -1
@@ -653,7 +653,7 @@ int paging_domain_init(struct domain *d)if (hap_enabled (d) )hap_domain_init (d);else - rc =shadow_domain_init (d, d->options); + rc =shadow_domain_init (d);return rc; }
xen/arch/x86/mm/shadow/common.c +4 -3
@@ -46,7 +46,7 @@ static void sh_clean_dirty_bitmap(struct domain *);/* Set up the shadow-specific parts of a domain struct at start of day. * Called for every domain from arch_domain_create() */ -int shadow_domain_init (struct domain *d,unsigned int domcr_flags) +int shadow_domain_init (struct domain *d) {static const struct log_dirty_ops sh_ops = { .enable = sh_enable_log_dirty,
@@ -62,7 +62,6 @@ int shadow_domain_init(struct domain *d, unsigned int domcr_flags)#if (SHADOW_OPTIMIZATIONS &SHOPT_OUT_OF_SYNC ) d->arch.paging.shadow.oos_active =0 ; - d->arch.paging.shadow.oos_off = domcr_flags & XEN_DOMCTL_CDF_oos_off;#endif d->arch.paging.shadow.pagetable_dying_op =0 ;
@@ -2523,11 +2522,13 @@ static void sh_update_paging_modes(struct vcpu *v)#if (SHADOW_OPTIMIZATIONS &SHOPT_OUT_OF_SYNC )/* We need to check that all the vcpus have paging enabled to * unsync PTs. */ -if (is_hvm_domain (d) && !d->arch.paging.shadow.oos_off ) +if ( !(d->options & XEN_DOMCTL_CDF_oos_off) ) {int pe =1 ;struct vcpu *vptr; +ASSERT (is_hvm_domain (d)); +for_each_vcpu (d, vptr) {if ( !hvm_paging_enabled (vptr) )
xen/arch/x86/mm/shadow/none.c +1 -1
@@ -18,7 +18,7 @@ static void _clean_dirty_bitmap(struct domain *d)ASSERT (is_pv_domain (d)); } -int shadow_domain_init (struct domain *d,unsigned int domcr_flags) +int shadow_domain_init (struct domain *d) {static const struct log_dirty_ops sh_none_ops = { .enable = _enable_log_dirty,
xen/common/domain.c +12 -4
@@ -313,11 +313,19 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } -if ( !(config->flags & XEN_DOMCTL_CDF_hvm_guest) && - (config->flags & XEN_DOMCTL_CDF_hap) ) +if ( !(config->flags & XEN_DOMCTL_CDF_hvm_guest) ) { -dprintk (XENLOG_INFO ,"HAP enabled for non-HVM guest\n" ); -return -EINVAL ; +if ( config->flags & XEN_DOMCTL_CDF_hap ) + { +dprintk (XENLOG_INFO ,"HAP enabled for non-HVM guest\n" ); +return -EINVAL ; + } + +/* +* It is only meaningful for XEN_DOMCTL_CDF_oos_off to be clear +* for HVM guests. +*/ + config->flags |= XEN_DOMCTL_CDF_oos_off; }return arch_sanitise_domain_config (config);
xen/include/asm-x86/domain.h +0 -1
@@ -115,7 +115,6 @@ struct shadow_domain {/* OOS */ bool_t oos_active; -bool_t oos_off;/* Has this domain ever used HVMOP_pagetable_dying? */ bool_t pagetable_dying_op;
xen/include/asm-x86/shadow.h +1 -1
@@ -49,7 +49,7 @@/* Set up the shadow-specific parts of a domain struct at start of day. * Called from paging_domain_init(). */ -int shadow_domain_init (struct domain *d,unsigned int domcr_flags); +int shadow_domain_init (struct domain *d);/* Setup the shadow-specific parts of a vcpu struct. It is called by * paging_vcpu_init() in paging.c */
Signature · 7 lines
2.20.1.2.gb21ebb671 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 3/10 domain: remove the 'is_xenstore' flag
In reply to Paul DurrantThis patch introduces a convenience macro, is_xenstore_domain(), which tests the domain 'options' directly and then uses that in place of the 'is_xenstore' flag.
---
Cc: Andrew Cooper <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Daniel De Graaf <[email protected]>
v2:
- Set 'disable_migrate' to true rather 1
--- xen/common/domain.c | 9 +++------ xen/common/domctl.c | 2 +- xen/include/xen/sched.h | 7 +++++-- xen/include/xsm/dummy.h | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-)
xen/common/domain.c +3 -6
@@ -363,7 +363,7 @@ struct domain *domain_create(domid_t domid,if ( hardware_domid <0 || hardware_domid >=DOMID_FIRST_RESERVED )panic ("The value of hardware_dom must be a valid domain ID\n" ); - d->disable_migrate =1 ; + d->disable_migrate = true; old_hwdom = hardware_domain; hardware_domain = d; }
@@ -442,11 +442,8 @@ struct domain *domain_create(domid_t domid,watchdog_domain_init (d); init_status |= INIT_watchdog; -if ( d->options & XEN_DOMCTL_CDF_xs_domain ) - { - d->is_xenstore =1 ; - d->disable_migrate =1 ; - } +if (is_xenstore_domain (d) ) + d->disable_migrate = true; d->iomem_caps =rangeset_new (d,"I/O Memory" , RANGESETF_prettyprint_hex); d->irq_caps =rangeset_new (d,"Interrupts" ,0 );
xen/common/domctl.c +1 -1
@@ -186,7 +186,7 @@ void getdomaininfo(struct domain *d, struct xen_domctl_getdomaininfo *info) (d->is_shut_down ? XEN_DOMINF_shutdown :0 ) | (d->controller_pause_count >0 ? XEN_DOMINF_paused :0 ) | (d->debugger_attached ? XEN_DOMINF_debugged :0 ) | - (d->is_xenstore ? XEN_DOMINF_xs_domain :0 ) | + (is_xenstore_domain (d) ? XEN_DOMINF_xs_domain :0 ) | (is_hvm_domain (d) ? XEN_DOMINF_hvm_guest :0 ) | d->shutdown_code << XEN_DOMINF_shutdownshift;
xen/include/xen/sched.h +5 -2
@@ -375,8 +375,6 @@ struct domainbool is_privileged;/* Can this guest access the Xen console? */ bool is_console; -/* Is this a xenstore domain (not dom0)? */ -bool is_xenstore;/* Non-migratable and non-restoreable? */ bool disable_migrate;/* Is this guest being debugged by dom0? */
@@ -978,6 +976,11 @@ static inline bool is_vcpu_online(const struct vcpu *v)return !test_bit (_VPF_down, &v->pause_flags); } +static inline bool is_xenstore_domain (const struct domain *d) +{ +return d->options & XEN_DOMCTL_CDF_xs_domain; +} +extern bool sched_smt_power_savings; extern enum cpufreq_controller {
xen/include/xsm/dummy.h +1 -1
@@ -79,7 +79,7 @@ static always_inline int xsm_default_action( {return 0 ;case XSM_XS_PRIV : -if ( src->is_xenstore ) +if (is_xenstore_domain (src) )return 0 ; }/* fall through */
Signature · 7 lines
2.20.1.2.gb21ebb671 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 4/10 x86/domain: remove the 's3_integrity' flag
In reply to Paul DurrantThe flag is not needed since the domain 'options' can now be tested directly.
---
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
v4:
- s/TBOOT/CONFIG_TBOOT/g
v3:
- Also sanitise the flag against CONFIG_TBOOT being set
--- xen/arch/x86/domain.c | 9 +++++++-- xen/arch/x86/setup.c | 2 +- xen/arch/x86/tboot.c | 2 +- xen/include/asm-x86/domain.h | 2 -- 4 files changed, 9 insertions(+), 6 deletions(-)
xen/arch/x86/domain.c +7 -2
@@ -466,6 +466,13 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +if ( (config->flags & XEN_DOMCTL_CDF_s3_integrity) && + !IS_ENABLED (CONFIG_TBOOT ) ) + { +dprintk (XENLOG_INFO ,"S3 integrity check not valid without CONFIG_TBOOT\n" ); +return -EINVAL ; + } +return 0 ; }
@@ -544,8 +551,6 @@ int arch_domain_create(struct domain *d, d->domain_id); } - d->arch.s3_integrity = config->flags & XEN_DOMCTL_CDF_s3_integrity; - emflags = config->arch.emulation_flags;if (is_hardware_domain (d) &&is_pv_domain (d) )
xen/arch/x86/setup.c +1 -1
@@ -696,7 +696,7 @@ void __init noreturn __start_xen(unsigned long mbi_p) .stop_bits = 1 };struct xen_domctl_createdomain dom0_cfg = { - .flags = XEN_DOMCTL_CDF_s3_integrity, + .flags =IS_ENABLED (CONFIG_TBOOT ) ? XEN_DOMCTL_CDF_s3_integrity :0 , .max_evtchn_port =-1 , .max_grant_frames = opt_max_grant_frames, .max_maptrack_frames = opt_max_maptrack_frames,
xen/arch/x86/tboot.c +1 -1
@@ -212,7 +212,7 @@ static void tboot_gen_domain_integrity(const uint8_t key[TB_KEY_SIZE],vmac_set_key ((uint8_t *)key, &ctx);for_each_domain ( d ) { -if ( !d->arch.s3_integrity ) +if ( !(d->options & XEN_DOMCTL_CDF_s3_integrity) )continue ;printk ("MACing Domain %u\n" , d->domain_id);
xen/include/asm-x86/domain.h +0 -2
@@ -295,8 +295,6 @@ struct arch_domainuint32_t pci_cf8;uint8_t cmos_idx; -bool_t s3_integrity; -union {struct pv_domain pv;struct hvm_domain hvm;
Signature · 7 lines
2.20.1.2.gb21ebb671 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 5/10 domain: introduce XEN_DOMCTL_CDF_iommu flag
In reply to Paul DurrantThis patch introduces a common domain creation flag to determine whether the domain is permitted to make use of the IOMMU. Currently the flag is always set (for both dom0 and domU) if the IOMMU is globally enabled (i.e. iommu_enabled == 1). sanitise_domain_config() is modified to reject the flag if !iommu_enabled.
A new helper function, is_iommu_enabled(), is added to test the flag and iommu_domain_init() will return immediately if !is_iommu_enabled(). This is slightly different to the previous behaviour based on !iommu_enabled where the call to arch_iommu_domain_init() was made regardless, however it appears that this call was only necessary to initialize the dt_devices list for ARM such that iommu_release_dt_devices() can be called unconditionally by domain_relinquish_resources(). Adding a simple check of is_iommu_enabled() into iommu_release_dt_devices() keeps this unconditional call working.
No functional change should be observed with this patch applied.
Subsequent patches will allow the toolstack to control whether use of the IOMMU is enabled for a domain.
NOTE: The introduction of the is_iommu_enabled() helper function might
seem excessive but its use is expected to increase with subsequent patches. Also, having iommu_domain_init() bail before calling arch_iommu_domain_init() is not strictly necessary, but I think the consequent addition of the call to is_iommu_enabled() in iommu_release_dt_devices() makes the code clearer.
---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html
v5:
- Move is_iommu_enabled() check into iommu_domain_init()
- Reject XEN_DOMCTL_CDF_iommu in sanitise_domain_config() if !iommu_enabled
- Use evaluate_nospec() in defintion of is_iommu_enabled()
--- tools/libxl/libxl_create.c | 8 ++++++++ xen/arch/arm/domain.c | 1 - xen/arch/arm/setup.c | 3 +++ xen/arch/x86/setup.c | 3 +++ xen/common/domain.c | 9 ++++++++- xen/drivers/passthrough/device_tree.c | 3 +++ xen/drivers/passthrough/iommu.c | 6 +++--- xen/include/public/domctl.h | 4 ++++ xen/include/xen/sched.h | 5 +++++ 9 files changed, 37 insertions(+), 5 deletions(-)
tools/libxl/libxl_create.c +8 -0
@@ -555,6 +555,7 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, .max_grant_frames = b_info->max_grant_frames, .max_maptrack_frames = b_info->max_maptrack_frames, }; +libxl_physinfo physinfo;if (info->type !=LIBXL_DOMAIN_TYPE_PV ) { create.flags |= XEN_DOMCTL_CDF_hvm_guest;
@@ -564,6 +565,13 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,libxl_defbool_val (info->oos) ?0 : XEN_DOMCTL_CDF_oos_off; } + rc =libxl_get_physinfo (ctx, &physinfo); +if (rc <0 ) + goto out; + +if (physinfo.cap_hvm_directio) + create.flags |= XEN_DOMCTL_CDF_iommu; +/* Ultimately, handle is an array of 16 uint8_t, same as uuid */ libxl_uuid_copy (ctx, (libxl_uuid *)&create.handle, &info->uuid);
xen/arch/arm/domain.c +0 -1
@@ -673,7 +673,6 @@ int arch_domain_create(struct domain *d,ASSERT (config !=NULL ); -/* p2m_init relies on some value initialized by the IOMMU subsystem */ if ( (rc =iommu_domain_init (d)) !=0 ) goto fail;
xen/arch/arm/setup.c +3 -0
@@ -914,6 +914,9 @@ void __init start_xen(unsigned long boot_phys_offset, dom0_cfg.arch.tee_type =tee_get_type (); dom0_cfg.max_vcpus =dom0_max_vcpus (); +if ( iommu_enabled ) + dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu; + dom0 =domain_create (0 , &dom0_cfg, true);if (IS_ERR (dom0) || (alloc_dom0_vcpu0 (dom0) ==NULL ) )panic ("Error creating domain 0\n" );
xen/arch/x86/setup.c +3 -0
@@ -1733,6 +1733,9 @@ void __init noreturn __start_xen(unsigned long mbi_p) } dom0_cfg.max_vcpus =dom0_max_vcpus (); +if ( iommu_enabled ) + dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu; +/* Create initial domain 0. */ dom0 =domain_create (get_initial_domain_id (), &dom0_cfg, !pv_shim);if (IS_ERR (dom0) || (alloc_dom0_vcpu0 (dom0) ==NULL ) )
xen/common/domain.c +8 -1
@@ -301,7 +301,8 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config) XEN_DOMCTL_CDF_hap | XEN_DOMCTL_CDF_s3_integrity | XEN_DOMCTL_CDF_oos_off | - XEN_DOMCTL_CDF_xs_domain) ) + XEN_DOMCTL_CDF_xs_domain | + XEN_DOMCTL_CDF_iommu) ) {dprintk (XENLOG_INFO ,"Unknown CDF flags %#x\n" , config->flags);return -EINVAL ;
@@ -328,6 +329,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config) config->flags |= XEN_DOMCTL_CDF_oos_off; } +if ( (config->flags & XEN_DOMCTL_CDF_iommu) && !iommu_enabled ) + { +dprintk (XENLOG_INFO ,"IOMMU is not enabled\n" ); +return -EINVAL ; + } +return arch_sanitise_domain_config (config); }
xen/drivers/passthrough/device_tree.c +3 -0
@@ -119,6 +119,9 @@ int iommu_release_dt_devices(struct domain *d)struct dt_device_node *dev, *_dev;int rc; +if ( !is_iommu_enabled (d) ) +return 0 ; +list_for_each_entry_safe (dev, _dev, &hd->dt_devices, domain_list) { rc =iommu_deassign_dt_device (d, dev);
xen/drivers/passthrough/iommu.c +3 -3
@@ -151,13 +151,13 @@ int iommu_domain_init(struct domain *d)struct domain_iommu *hd =dom_iommu (d);int ret =0 ; +if ( !is_iommu_enabled (d) ) +return 0 ; + ret =arch_iommu_domain_init (d);if ( ret )return ret; -if ( !iommu_enabled ) -return 0 ; - hd->platform_ops =iommu_get_ops ();return hd->platform_ops->init (d); }
xen/include/public/domctl.h +4 -0
@@ -64,6 +64,10 @@ struct xen_domctl_createdomain {/* Is this a xenstore domain? */ #define _XEN_DOMCTL_CDF_xs_domain 4#define XEN_DOMCTL_CDF_xs_domain (1U<<_XEN_DOMCTL_CDF_xs_domain) +/* Should this domain be permitted to use the IOMMU? */ +#define _XEN_DOMCTL_CDF_iommu 5 +#define XEN_DOMCTL_CDF_iommu (1U<<_XEN_DOMCTL_CDF_iommu) +uint32_t flags;/*
xen/include/xen/sched.h +5 -0
@@ -981,6 +981,11 @@ static inline bool is_xenstore_domain(const struct domain *d)return d->options & XEN_DOMCTL_CDF_xs_domain; } +static inline bool is_iommu_enabled (const struct domain *d) +{ +return evaluate_nospec (d->options & XEN_DOMCTL_CDF_iommu); +} +extern bool sched_smt_power_savings; extern enum cpufreq_controller {
Signature · 7 lines
2.20.1.2.gb21ebb671 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v5 05/10] domain: introduce XEN_DOMCTL_CDF_iommu flag
Hi Paul,
Quoted 73 lines · Paul Durrant
This patch introduces a common domain creation flag to determine whether the domain is permitted to make use of the IOMMU. Currently the flag is always set (for both dom0 and domU) if the IOMMU is globally enabled (i.e.
iommu_enabled== 1).sanitise_domain_config()is modified to reject the flag if !iommu_enabled.A new helper function,
is_iommu_enabled(), is added to test the flag andiommu_domain_init()will return immediately if !is_iommu_enabled(). This is slightly different to the previous behaviour based on !iommu_enabled where the call toarch_iommu_domain_init()was made regardless, however it appears that this call was only necessary to initialize thedt_deviceslist for ARM such thatiommu_release_dt_devices()can be called unconditionally bydomain_relinquish_resources(). Adding a simple check ofis_iommu_enabled()intoiommu_release_dt_devices()keeps this unconditional call working.No functional change should be observed with this patch applied.
Subsequent patches will allow the toolstack to control whether use of the IOMMU is enabled for a domain.
NOTE: The introduction of the
is_iommu_enabled()helper function might
seem excessive but its use is expected to increase with subsequent patches. Also, havingiommu_domain_init()bail before callingarch_iommu_domain_init()is not strictly necessary, but I think the consequent addition of the call tois_iommu_enabled()iniommu_release_dt_devices()makes the code clearer.Signed-off-by Paul Durrant <[email protected]>---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html
v5:
- Moveis_iommu_enabled()check intoiommu_domain_init()
- RejectXEN_DOMCTL_CDF_iommuinsanitise_domain_config()if !iommu_enabled
- Useevaluate_nospec()in defintion ofis_iommu_enabled()--- tools/libxl/libxl_create.c | 8 ++++++++ xen/arch/arm/domain.c | 1 - xen/arch/arm/setup.c | 3 +++ xen/arch/x86/setup.c | 3 +++ xen/common/domain.c | 9 ++++++++- xen/drivers/passthrough/device_tree.c | 3 +++ xen/drivers/passthrough/iommu.c | 6 +++--- xen/include/public/domctl.h | 4 ++++ xen/include/xen/sched.h | 5 +++++ 9 files changed, 37 insertions(+), 5 deletions(-)tools/libxl/libxl_create.c +1
-0@@ -555,6 +555,7 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, .max_grant_frames = b_info->max_grant_frames, .max_maptrack_frames = b_info->max_maptrack_frames, }; +libxl_physinfo physinfo;if (info->type !=LIBXL_DOMAIN_TYPE_PV ) { create.flags |= XEN_DOMCTL_CDF_hvm_guest;
tools/libxl/libxl_create.c int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, · line 573
Quoted 9 lines
tools/libxl/libxl_create.c +6
-0@@ -564,6 +565,13 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,libxl_defbool_val (info->oos) ?0 : XEN_DOMCTL_CDF_oos_off; } + rc =libxl_get_physinfo (ctx, &physinfo); +if (rc <0 ) + goto out; + +if (physinfo.cap_hvm_directio) + create.flags |= XEN_DOMCTL_CDF_iommu;
This is not going to work well on Arm as XEN_SYSCTL_PHYSCAP_directio is never set.
xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 676
Quoted 13 lines
tools/libxl/libxl_create.c +1
-0+/* Ultimately, handle is an array of 16 uint8_t, same as uuid */ libxl_uuid_copy (ctx, (libxl_uuid *)&create.handle, &info->uuid);xen/arch/arm/domain.c +0
-1@@ -673,7 +673,6 @@ int arch_domain_create(struct domain *d,ASSERT (config !=NULL ); -/* p2m_init relies on some value initialized by the IOMMU subsystem */
Even with this patch, I still think iommu_domain_init() is required before calling p2m_init(). For instance, this function will set the features flag.
xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 677
Quoted 2 lines
xen/arch/arm/domain.c +0
-0if ( (rc =iommu_domain_init (d)) !=0 ) goto fail;
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v5 05/10] domain: introduce XEN_DOMCTL_CDF_iommu flag
Quoted 90 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 14 August 2019 15:00
To: Paul Durrant <[email protected]>; [email protected]
Cc: Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Anthony Perard <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [PATCH v5 05/10] domain: introduceXEN_DOMCTL_CDF_iommuflagHi Paul,
Quoted 76 lines · Paul Durrant
This patch introduces a common domain creation flag to determine whether the domain is permitted to make use of the IOMMU. Currently the flag is always set (for both dom0 and domU) if the IOMMU is globally enabled (i.e.
iommu_enabled== 1).sanitise_domain_config()is modified to reject the flag if !iommu_enabled.A new helper function,
is_iommu_enabled(), is added to test the flag andiommu_domain_init()will return immediately if !is_iommu_enabled(). This is slightly different to the previous behaviour based on !iommu_enabled where the call toarch_iommu_domain_init()was made regardless, however it appears that this call was only necessary to initialize thedt_deviceslist for ARM such thatiommu_release_dt_devices()can be called unconditionally bydomain_relinquish_resources(). Adding a simple check ofis_iommu_enabled()intoiommu_release_dt_devices()keeps this unconditional call working.No functional change should be observed with this patch applied.
Subsequent patches will allow the toolstack to control whether use of the IOMMU is enabled for a domain.
NOTE: The introduction of the
is_iommu_enabled()helper function might
seem excessive but its use is expected to increase with subsequent patches. Also, havingiommu_domain_init()bail before callingarch_iommu_domain_init()is not strictly necessary, but I think the consequent addition of the call tois_iommu_enabled()iniommu_release_dt_devices()makes the code clearer.Signed-off-by Paul Durrant <[email protected]>---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html
v5:
- Moveis_iommu_enabled()check intoiommu_domain_init()
- RejectXEN_DOMCTL_CDF_iommuinsanitise_domain_config()if !iommu_enabled
- Useevaluate_nospec()in defintion ofis_iommu_enabled()--- tools/libxl/libxl_create.c | 8 ++++++++ xen/arch/arm/domain.c | 1 - xen/arch/arm/setup.c | 3 +++ xen/arch/x86/setup.c | 3 +++ xen/common/domain.c | 9 ++++++++- xen/drivers/passthrough/device_tree.c | 3 +++ xen/drivers/passthrough/iommu.c | 6 +++--- xen/include/public/domctl.h | 4 ++++ xen/include/xen/sched.h | 5 +++++ 9 files changed, 37 insertions(+), 5 deletions(-)tools/libxl/libxl_create.c +1
-0@@ -555,6 +555,7 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, .max_grant_frames = b_info->max_grant_frames, .max_maptrack_frames = b_info->max_maptrack_frames, }; +libxl_physinfo physinfo;if (info->type !=LIBXL_DOMAIN_TYPE_PV ) { create.flags |= XEN_DOMCTL_CDF_hvm_guest;@@ -564,6 +565,13 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,libxl_defbool_val (info->oos) ?0 : XEN_DOMCTL_CDF_oos_off; }
Quoted 8 lines
tools/libxl/libxl_create.c int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, · line 573
Quoted 6 lines
tools/libxl/libxl_create.c +6
-0+ rc =libxl_get_physinfo (ctx, &physinfo); +if (rc <0 ) + goto out; + +if (physinfo.cap_hvm_directio) + create.flags |= XEN_DOMCTL_CDF_iommu;This is not going to work well on Arm as
XEN_SYSCTL_PHYSCAP_directiois never set.
Oh, that's true. I need to pull that into the common sysctl handler. It also looks like XEN_SYSCTL_PHYSCAP_hvm should always be set for too, but ARMs arch_do_physinfo() is completely empty at the moment.
Quoted 11 lines
Quoted 11 lines
tools/libxl/libxl_create.c +1
-0+/* Ultimately, handle is an array of 16 uint8_t, same as uuid */ libxl_uuid_copy (ctx, (libxl_uuid *)&create.handle, &info->uuid);xen/arch/arm/domain.c +0
-0@@ -673,7 +673,6 @@ int arch_domain_create(struct domain *d,
Quoted 6 lines
xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 676
Quoted 3 lines
xen/arch/arm/domain.c +0
-1ASSERT (config !=NULL ); -/* p2m_init relies on some value initialized by the IOMMU subsystem */ Even with this patch, I still think
iommu_domain_init()is required before callingp2m_init(). For instance, this function will set the features flag.
Oh, yes, I see that's in the SMMU code. I'll leave the comment in place... I thought it was only there because of arch_iommu_domain_init() being called unconditionally before (regardless of iommu_enabled).
Paul
Quoted 8 lines
xen/arch/arm/domain.c int arch_domain_create(struct domain *d, · line 677
Quoted 3 lines
xen/arch/arm/domain.c +0
-0if ( (rc =iommu_domain_init (d)) !=0 ) goto fail;Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v5 05/10] domain: introduce XEN_DOMCTL_CDF_iommu flag
Hi Paul,
Quoted 99 lines · Paul Durrant
Quoted 90 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 14 August 2019 15:00
To: Paul Durrant <[email protected]>; [email protected]
Cc: Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Anthony Perard <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [PATCH v5 05/10] domain: introduceXEN_DOMCTL_CDF_iommuflagHi Paul,
Quoted 76 lines · Paul Durrant
This patch introduces a common domain creation flag to determine whether the domain is permitted to make use of the IOMMU. Currently the flag is always set (for both dom0 and domU) if the IOMMU is globally enabled (i.e.
iommu_enabled== 1).sanitise_domain_config()is modified to reject the flag if !iommu_enabled.A new helper function,
is_iommu_enabled(), is added to test the flag andiommu_domain_init()will return immediately if !is_iommu_enabled(). This is slightly different to the previous behaviour based on !iommu_enabled where the call toarch_iommu_domain_init()was made regardless, however it appears that this call was only necessary to initialize thedt_deviceslist for ARM such thatiommu_release_dt_devices()can be called unconditionally bydomain_relinquish_resources(). Adding a simple check ofis_iommu_enabled()intoiommu_release_dt_devices()keeps this unconditional call working.No functional change should be observed with this patch applied.
Subsequent patches will allow the toolstack to control whether use of the IOMMU is enabled for a domain.
NOTE: The introduction of the
is_iommu_enabled()helper function might
seem excessive but its use is expected to increase with subsequent patches. Also, havingiommu_domain_init()bail before callingarch_iommu_domain_init()is not strictly necessary, but I think the consequent addition of the call tois_iommu_enabled()iniommu_release_dt_devices()makes the code clearer.Signed-off-by Paul Durrant <[email protected]>---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html
v5:
- Moveis_iommu_enabled()check intoiommu_domain_init()
- RejectXEN_DOMCTL_CDF_iommuinsanitise_domain_config()if !iommu_enabled
- Useevaluate_nospec()in defintion ofis_iommu_enabled()--- tools/libxl/libxl_create.c | 8 ++++++++ xen/arch/arm/domain.c | 1 - xen/arch/arm/setup.c | 3 +++ xen/arch/x86/setup.c | 3 +++ xen/common/domain.c | 9 ++++++++- xen/drivers/passthrough/device_tree.c | 3 +++ xen/drivers/passthrough/iommu.c | 6 +++--- xen/include/public/domctl.h | 4 ++++ xen/include/xen/sched.h | 5 +++++ 9 files changed, 37 insertions(+), 5 deletions(-)tools/libxl/libxl_create.c +1
-0@@ -555,6 +555,7 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, .max_grant_frames = b_info->max_grant_frames, .max_maptrack_frames = b_info->max_maptrack_frames, }; +libxl_physinfo physinfo;if (info->type !=LIBXL_DOMAIN_TYPE_PV ) { create.flags |= XEN_DOMCTL_CDF_hvm_guest;@@ -564,6 +565,13 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,libxl_defbool_val (info->oos) ?0 : XEN_DOMCTL_CDF_oos_off; }Quoted 8 lines
tools/libxl/libxl_create.c int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, · line 573
Quoted 6 lines
tools/libxl/libxl_create.c +6
-0+ rc =libxl_get_physinfo (ctx, &physinfo); +if (rc <0 ) + goto out; + +if (physinfo.cap_hvm_directio) + create.flags |= XEN_DOMCTL_CDF_iommu;This is not going to work well on Arm as
XEN_SYSCTL_PHYSCAP_directiois never set.
Quoted 1 line
Oh, that's true. I need to pull that into the common sysctl handler. It also looks like
XEN_SYSCTL_PHYSCAP_hvmshould always be set for too, but ARMsarch_do_physinfo()is completely empty at the moment.
Looking at the header (include/public/sysctl.h) it is pretty clear that the two PHYSCAP are x86 specific.
Even if Arm guests are based on the same concept as HVM/PVH guest. They are just called "Arm guest". We have always taken action so far to say they are HVM/PVH guest when presenting to the user. For instance, we strongly recommend to have the guest type in your configuration.
So I am not entirely sure we want to expose the capability for Arm. Otherwise a user may think it is fine to use "hvm" guest type.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v5 05/10] domain: introduce XEN_DOMCTL_CDF_iommu flag
Quoted 121 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 14 August 2019 18:15
To: Paul Durrant <[email protected]>; [email protected]
Cc: Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Anthony Perard <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [PATCH v5 05/10] domain: introduceXEN_DOMCTL_CDF_iommuflagHi Paul,
Quoted 56 lines · Paul Durrant
Quoted 56 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 14 August 2019 15:00
To: Paul Durrant <[email protected]>; [email protected]
Cc: Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Anthony Perard <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [PATCH v5 05/10] domain: introduceXEN_DOMCTL_CDF_iommuflagHi Paul,
Quoted 42 lines · Paul Durrant
This patch introduces a common domain creation flag to determine whether the domain is permitted to make use of the IOMMU. Currently the flag is always set (for both dom0 and domU) if the IOMMU is globally enabled (i.e.
iommu_enabled== 1).sanitise_domain_config()is modified to reject the flag if !iommu_enabled.A new helper function,
is_iommu_enabled(), is added to test the flag andiommu_domain_init()will return immediately if !is_iommu_enabled(). This is slightly different to the previous behaviour based on !iommu_enabled where the call toarch_iommu_domain_init()was made regardless, however it appears that this call was only necessary to initialize thedt_deviceslist for ARM such thatiommu_release_dt_devices()can be called unconditionally bydomain_relinquish_resources(). Adding a simple check ofis_iommu_enabled()intoiommu_release_dt_devices()keeps this unconditional call working.No functional change should be observed with this patch applied.
Subsequent patches will allow the toolstack to control whether use of the IOMMU is enabled for a domain.
NOTE: The introduction of the
is_iommu_enabled()helper function might
seem excessive but its use is expected to increase with subsequent patches. Also, havingiommu_domain_init()bail before callingarch_iommu_domain_init()is not strictly necessary, but I think the consequent addition of the call tois_iommu_enabled()iniommu_release_dt_devices()makes the code clearer.Signed-off-by Paul Durrant <[email protected]>---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>Quoted 1 line
Quoted 1 line
Quoted 1 line
Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-
07/msg02267.html
Quoted 42 lines
Quoted 33 lines
Quoted 33 lines
v5:
- Moveis_iommu_enabled()check intoiommu_domain_init()
- RejectXEN_DOMCTL_CDF_iommuinsanitise_domain_config()if !iommu_enabled
- Useevaluate_nospec()in defintion ofis_iommu_enabled()--- tools/libxl/libxl_create.c | 8 ++++++++ xen/arch/arm/domain.c | 1 - xen/arch/arm/setup.c | 3 +++ xen/arch/x86/setup.c | 3 +++ xen/common/domain.c | 9 ++++++++- xen/drivers/passthrough/device_tree.c | 3 +++ xen/drivers/passthrough/iommu.c | 6 +++--- xen/include/public/domctl.h | 4 ++++ xen/include/xen/sched.h | 5 +++++ 9 files changed, 37 insertions(+), 5 deletions(-)tools/libxl/libxl_create.c +1
-0@@ -555,6 +555,7 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, .max_grant_frames = b_info->max_grant_frames, .max_maptrack_frames = b_info->max_maptrack_frames, }; +libxl_physinfo physinfo;if (info->type !=LIBXL_DOMAIN_TYPE_PV ) { create.flags |= XEN_DOMCTL_CDF_hvm_guest;@@ -564,6 +565,13 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,libxl_defbool_val (info->oos) ?0 : XEN_DOMCTL_CDF_oos_off; }Quoted 8 lines
tools/libxl/libxl_create.c int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config, · line 573
Quoted 6 lines
tools/libxl/libxl_create.c +6
-0+ rc =libxl_get_physinfo (ctx, &physinfo); +if (rc <0 ) + goto out; + +if (physinfo.cap_hvm_directio) + create.flags |= XEN_DOMCTL_CDF_iommu;This is not going to work well on Arm as
XEN_SYSCTL_PHYSCAP_directiois never set.Quoted 1 line
Oh, that's true. I need to pull that into the common sysctl handler. It also looks like
XEN_SYSCTL_PHYSCAP_hvmshould always be set for too, but ARMsarch_do_physinfo()is completely empty at the moment.Looking at the header (
include/public/sysctl.h) it is pretty clear that the two PHYSCAP are x86 specific.
Quoted 7 lines
Even if Arm guests are based on the same concept as HVM/PVH guest. They are just called "Arm guest". We have always taken action so far to say they are HVM/PVH guest when presenting to the user. For instance, we strongly recommend to have the guest type in your configuration.
So I am not entirely sure we want to expose the capability for Arm. Otherwise a user may think it is fine to use "hvm" guest type.
Ok, yes I see the directio flag is listed as x86 too... I guess I'll have to forget this mechanism and hardcode the flag to the global iommu_enabled flag inside the hypervisor.
Paul
Quoted 5 lines
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 6/10 use is_iommu_enabled() where appropriate...
In reply to Paul Durrant...rather than testing the global iommu_enabled flag and ops pointer.
Now that there is a per-domain flag indicating whether the domain is permitted to use the IOMMU (which determines whether the ops pointer will be set), many tests of the global iommu_enabled flag and ops pointer can be translated into tests of the per-domain flag. Some of the other tests of purely the global iommu_enabled flag can also be translated into tests of the per-domain flag.
NOTE: The comment in iommu_share_p2m_table() is also fixed; need_iommu()
disappeared some time ago. Also, whilst the style of the 'if' in flask_iommu_resource_use_perm() is fixed, I have not translated any instances of u32 into uint32_t to keep consistency. IMO such a translation would be better done globally for the source module in a separate patch.
---
Cc: Stefano Stabellini <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Jun Nakajima <[email protected]>
Cc: Kevin Tian <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Suravee Suthikulpanit <[email protected]>
Cc: Brian Woods <[email protected]>
Cc: Daniel De Graaf <[email protected]>
Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html
This patch should be applied after Alexandru Isaila's 'Clean iommu_hap_pt_share enabled code' patch:
https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg01708.html
v5:
- Fix logic in ARM p2m_init()
- Make iommu_do_domctl() return -EOPNOTSUPP rather than -ENOSYS if the IOMMU is not enabled
- Fix test in pci_enable_acs()
- Fix test in flask_iommu_resource_use_perm()
--- xen/arch/arm/p2m.c | 2 +- xen/arch/x86/dom0_build.c | 2 +- xen/arch/x86/domctl.c | 4 +-- xen/arch/x86/hvm/hvm.c | 6 ++--- xen/arch/x86/hvm/vioapic.c | 2 +- xen/arch/x86/hvm/vmx/vmcs.c | 2 +- xen/arch/x86/hvm/vmx/vmx.c | 2 +- xen/arch/x86/mm/p2m-ept.c | 4 +-- xen/drivers/passthrough/amd/iommu_guest.c | 2 +- xen/drivers/passthrough/device_tree.c | 4 +-- xen/drivers/passthrough/io.c | 8 +++--- xen/drivers/passthrough/iommu.c | 31 ++++++++++------------- xen/drivers/passthrough/pci.c | 16 ++++++------ xen/drivers/passthrough/vtd/iommu.c | 2 +- xen/drivers/passthrough/vtd/x86/hvm.c | 2 +- xen/drivers/passthrough/x86/iommu.c | 2 +- xen/xsm/flask/hooks.c | 18 ++++++------- 17 files changed, 53 insertions(+), 56 deletions(-)
xen/arch/arm/p2m.c +1 -1
@@ -1531,7 +1531,7 @@ int p2m_init(struct domain *d)* shared with the CPU, Xen has to make sure that the PT changes have * reached the memory */ - p2m->clean_pte = iommu_enabled && + p2m->clean_pte =is_iommu_enabled (d) && !iommu_has_feature (d,IOMMU_FEAT_COHERENT_WALK ); rc =p2m_alloc_table (d);
xen/arch/x86/dom0_build.c +1 -1
@@ -356,7 +356,7 @@ unsigned long __init dom0_compute_nr_pages( avail -= d->max_vcpus -1 ;/* Reserve memory for iommu_dom0_init() (rough estimate). */ -if ( iommu_enabled ) +if (is_iommu_enabled (d) ) {unsigned int s;
xen/arch/x86/domctl.c +2 -2
@@ -715,7 +715,7 @@ long arch_do_domctl(break ; ret = -ESRCH ; -if ( iommu_enabled ) +if (is_iommu_enabled (d) ) {pcidevs_lock (); ret =pt_irq_create_bind (d, bind);
@@ -744,7 +744,7 @@ long arch_do_domctl(if ( ret )break ; -if ( iommu_enabled ) +if (is_iommu_enabled (d) ) {pcidevs_lock (); ret =pt_irq_destroy_bind (d, bind);
xen/arch/x86/hvm/hvm.c +3 -3
@@ -465,7 +465,7 @@ void hvm_migrate_timers(struct vcpu *v)void hvm_migrate_pirq (struct hvm_pirq_dpci *pirq_dpci,const struct vcpu *v) { -ASSERT (iommu_enabled && +ASSERT (is_iommu_enabled (v->domain) && (is_hardware_domain (v->domain) ||hvm_domain_irq (v->domain)->dpci)); if ( (pirq_dpci->flags & HVM_IRQ_DPCI_MACH_MSI) &&
@@ -496,7 +496,7 @@ void hvm_migrate_pirqs(struct vcpu *v) {struct domain *d = v->domain; -if ( !iommu_enabled || !hvm_domain_irq (d)->dpci ) +if ( !is_iommu_enabled (d) || !hvm_domain_irq (d)->dpci )return ;spin_lock (&d->event_lock);
@@ -2264,7 +2264,7 @@ int hvm_set_cr0(unsigned long value, bool may_defer) }if ( ((value ^ old_value) &X86_CR0_CD ) && - iommu_enabled && hvm_funcs.handle_cd && +is_iommu_enabled (d) && hvm_funcs.handle_cd && (!rangeset_is_empty (d->iomem_caps) || !rangeset_is_empty (d->arch.ioport_caps) ||has_arch_pdevs (d)) )
xen/arch/x86/hvm/vioapic.c +1 -1
@@ -536,7 +536,7 @@ void vioapic_update_EOI(struct domain *d, u8 vector) ent->fields.remote_irr =0 ; -if ( iommu_enabled ) +if (is_iommu_enabled (d) ) {spin_unlock (&d->arch.hvm.irq_lock);hvm_dpci_eoi (d, vioapic->base_gsi + pin, ent);
xen/arch/x86/hvm/vmx/vmcs.c +1 -1
@@ -1087,7 +1087,7 @@ static int construct_vmcs(struct vcpu *v)vmx_clear_msr_intercept (v,MSR_IA32_SYSENTER_CS ,VMX_MSR_RW );vmx_clear_msr_intercept (v,MSR_IA32_SYSENTER_ESP ,VMX_MSR_RW );vmx_clear_msr_intercept (v,MSR_IA32_SYSENTER_EIP ,VMX_MSR_RW ); -if (paging_mode_hap (d) && (!iommu_enabled || iommu_snoop) ) +if (paging_mode_hap (d) && (!is_iommu_enabled (d) || iommu_snoop) )vmx_clear_msr_intercept (v,MSR_IA32_CR_PAT ,VMX_MSR_RW );if ( (vmexit_ctl &VM_EXIT_CLEAR_BNDCFGS ) && (vmentry_ctl &VM_ENTRY_LOAD_BNDCFGS ) )
xen/arch/x86/hvm/vmx/vmx.c +1 -1
@@ -1210,7 +1210,7 @@ static void vmx_handle_cd(struct vcpu *v, unsigned long value) { v->arch.hvm.cache_mode =NORMAL_CACHE_MODE ;vmx_set_guest_pat (v, *pat); -if ( !iommu_enabled || iommu_snoop ) +if ( !is_iommu_enabled (v->domain) || iommu_snoop )vmx_clear_msr_intercept (v,MSR_IA32_CR_PAT ,VMX_MSR_RW );hvm_asid_flush_vcpu (v);/* no need to flush cache */ }
xen/arch/x86/mm/p2m-ept.c +2 -2
@@ -260,7 +260,7 @@ static bool_t ept_split_super_page(struct p2m_domain *p2m, *epte = *ept_entry; epte->sp = (level >1 ); epte->mfn += i * trunk; - epte->snp = (iommu_enabled && iommu_snoop); + epte->snp =is_iommu_enabled (p2m->domain) && iommu_snoop; epte->suppress_ve =1 ;ept_p2m_type_to_flags (p2m, epte, epte->sa_p2mt, epte->access);
@@ -766,7 +766,7 @@ ept_set_entry(struct p2m_domain *p2m, gfn_t gfn_, mfn_t mfn, new_entry.sp = !!i; new_entry.sa_p2mt = p2mt; new_entry.access = p2ma; - new_entry.snp = (iommu_enabled && iommu_snoop); + new_entry.snp =is_iommu_enabled (d) && iommu_snoop;/* the caller should take care of the previous page */ new_entry.mfn =mfn_x (mfn);
xen/drivers/passthrough/amd/iommu_guest.c +1 -1
@@ -821,7 +821,7 @@ int guest_iommu_init(struct domain* d)struct guest_iommu *iommu;struct domain_iommu *hd =dom_iommu (d); -if ( !is_hvm_domain (d) || !iommu_enabled || !iommuv2_enabled || +if ( !is_hvm_domain (d) || !is_iommu_enabled (d) || !iommuv2_enabled || !has_viommu (d) )return 0 ;
xen/drivers/passthrough/device_tree.c +2 -2
@@ -29,7 +29,7 @@ int iommu_assign_dt_device(struct domain *d, struct dt_device_node *dev)int rc = -EBUSY ;struct domain_iommu *hd =dom_iommu (d); -if ( !iommu_enabled || !hd->platform_ops ) +if ( !is_iommu_enabled (d) )return -EINVAL ;if ( !dt_device_is_protected (dev) )
@@ -71,7 +71,7 @@ int iommu_deassign_dt_device(struct domain *d, struct dt_device_node *dev)const struct domain_iommu *hd =dom_iommu (d);int rc; -if ( !iommu_enabled || !hd->platform_ops ) +if ( !is_iommu_enabled (d) )return -EINVAL ;if ( !dt_device_is_protected (dev) )
xen/drivers/passthrough/io.c +4 -4
@@ -434,7 +434,7 @@ int pt_irq_create_bind(if ( vcpu ) pirq_dpci->gmsi.posted = true; } -if ( vcpu && iommu_enabled ) +if ( vcpu &&is_iommu_enabled (d) )hvm_migrate_pirq (pirq_dpci, vcpu);/* Use interrupt posting if it is supported. */
@@ -817,7 +817,7 @@ int hvm_do_IRQ_dpci(struct domain *d, struct pirq *pirq)ASSERT (is_hvm_domain (d)); -if ( !iommu_enabled || (!is_hardware_domain (d) && !dpci) || +if ( !is_iommu_enabled (d) || (!is_hardware_domain (d) && !dpci) || !pirq_dpci || !(pirq_dpci->flags &HVM_IRQ_DPCI_MAPPED ) )return 0 ;
@@ -869,7 +869,7 @@ static int _hvm_dpci_msi_eoi(struct domain *d,void hvm_dpci_msi_eoi (struct domain *d,int vector) { -if ( !iommu_enabled || +if ( !is_iommu_enabled (d) || (!hvm_domain_irq (d)->dpci && !is_hardware_domain (d)) )return ;
@@ -1001,7 +1001,7 @@ void hvm_dpci_eoi(struct domain *d, unsigned int guest_gsi,const struct hvm_irq_dpci *hvm_irq_dpci;const struct hvm_girq_dpci_mapping *girq; -if ( !iommu_enabled ) +if ( !is_iommu_enabled (d) )return ;if (is_hardware_domain (d) )
xen/drivers/passthrough/iommu.c +14 -17
@@ -179,7 +179,7 @@ void __hwdom_init iommu_hwdom_init(struct domain *d)check_hwdom_reqs (d); -if ( !iommu_enabled ) +if ( !is_iommu_enabled (d) )return ;register_keyhandler ('o' , &iommu_dump_p2m_table,"dump iommu p2m table" ,0 );
@@ -284,7 +284,7 @@ int iommu_construct(struct domain *d)void iommu_domain_destroy (struct domain *d) { -if ( !iommu_enabled || !dom_iommu (d)->platform_ops ) +if ( !is_iommu_enabled (d) )return ;iommu_teardown (d);
@@ -300,7 +300,7 @@ int iommu_map(struct domain *d, dfn_t dfn, mfn_t mfn,unsigned long i;int rc =0 ; -if ( !iommu_enabled || !hd->platform_ops ) +if ( !is_iommu_enabled (d) )return 0 ;ASSERT (IS_ALIGNED (dfn_x (dfn), (1ul << page_order)));
@@ -360,7 +360,7 @@ int iommu_unmap(struct domain *d, dfn_t dfn, unsigned int page_order,unsigned long i;int rc =0 ; -if ( !iommu_enabled || !hd->platform_ops ) +if ( !is_iommu_enabled (d) )return 0 ;ASSERT (IS_ALIGNED (dfn_x (dfn), (1ul << page_order)));
@@ -413,7 +413,7 @@ int iommu_lookup_page(struct domain *d, dfn_t dfn, mfn_t *mfn, {const struct domain_iommu *hd =dom_iommu (d); -if ( !iommu_enabled || !hd->platform_ops || !hd->platform_ops->lookup_page ) +if ( !is_iommu_enabled (d) || !hd->platform_ops->lookup_page )return -EOPNOTSUPP ;return iommu_call (hd->platform_ops, lookup_page, d, dfn, mfn, flags);
@@ -442,8 +442,8 @@ int iommu_iotlb_flush(struct domain *d, dfn_t dfn, unsigned int page_count,const struct domain_iommu *hd =dom_iommu (d);int rc; -if ( !iommu_enabled || !hd->platform_ops || - !hd->platform_ops->iotlb_flush || !page_count || !flush_flags ) +if ( !is_iommu_enabled (d) || !hd->platform_ops->iotlb_flush || + !page_count || !flush_flags )return 0 ;if (dfn_eq (dfn,INVALID_DFN ) )
@@ -470,8 +470,8 @@ int iommu_iotlb_flush_all(struct domain *d, unsigned int flush_flags)const struct domain_iommu *hd =dom_iommu (d);int rc; -if ( !iommu_enabled || !hd->platform_ops || - !hd->platform_ops->iotlb_flush_all || !flush_flags ) +if ( !is_iommu_enabled (d) || !hd->platform_ops->iotlb_flush_all || + !flush_flags )return 0 ;/*
@@ -556,8 +556,8 @@ int iommu_do_domctl( {int ret = -ENODEV ; -if ( !iommu_enabled ) -return -ENOSYS ; +if ( !is_iommu_enabled (d) ) +return -EOPNOTSUPP ;#ifdef CONFIG_HAS_PCI ret =iommu_do_pci_domctl (domctl, d, u_domctl);
@@ -576,9 +576,9 @@ void iommu_share_p2m_table(struct domain* d)ASSERT (hap_enabled (d));/* * iommu_use_hap_pt(d) cannot be used here because during domain -* construction need_iommu(d) will always return false here. +* construction has_iommu_pt(d) will always return false here. */ -if ( iommu_enabled && iommu_hap_pt_share ) +if (is_iommu_enabled (d) && iommu_hap_pt_share )iommu_get_ops ()->share_p2m (d); }
@@ -608,10 +608,7 @@ int iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)bool_t iommu_has_feature (struct domain *d,enum iommu_feature feature) { -if ( !iommu_enabled ) -return 0 ; - -return test_bit (feature,dom_iommu (d)->features); +return is_iommu_enabled (d) &&test_bit (feature,dom_iommu (d)->features); }static void iommu_dump_p2m_table (unsigned char key)
xen/drivers/passthrough/pci.c +8 -8
@@ -595,7 +595,7 @@ static void pci_enable_acs(struct pci_dev *pdev)u16 cap, ctrl, seg = pdev->seg;u8 bus = pdev->bus; -if ( !iommu_enabled ) +if ( !is_iommu_enabled (pdev->domain) )return ; pos =pci_find_ext_capability (seg, bus, pdev->devfn,PCI_EXT_CAP_ID_ACS );
@@ -864,7 +864,7 @@ static int pci_clean_dpci_irqs(struct domain *d) {struct hvm_irq_dpci *hvm_irq_dpci =NULL ; -if ( !iommu_enabled ) +if ( !is_iommu_enabled (d) )return 0 ;if ( !is_hvm_domain (d) )
@@ -1333,7 +1333,7 @@ static int iommu_add_device(struct pci_dev *pdev)ASSERT (pcidevs_locked ()); hd =dom_iommu (pdev->domain); -if ( !iommu_enabled || !hd->platform_ops ) +if ( !is_iommu_enabled (pdev->domain) )return 0 ; rc = hd->platform_ops->add_device (pdev->devfn,pci_to_dev (pdev));
@@ -1362,7 +1362,7 @@ static int iommu_enable_device(struct pci_dev *pdev)ASSERT (pcidevs_locked ()); hd =dom_iommu (pdev->domain); -if ( !iommu_enabled || !hd->platform_ops || +if ( !is_iommu_enabled (pdev->domain) || !hd->platform_ops->enable_device )return 0 ;
@@ -1378,7 +1378,7 @@ static int iommu_remove_device(struct pci_dev *pdev)return -EINVAL ; hd =dom_iommu (pdev->domain); -if ( !iommu_enabled || !hd->platform_ops ) +if ( !is_iommu_enabled (pdev->domain) )return 0 ;for ( devfn = pdev->devfn ; pdev->phantom_stride; )
@@ -1421,7 +1421,7 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag)struct pci_dev *pdev;int rc =0 ; -if ( !iommu_enabled || !hd->platform_ops ) +if ( !is_iommu_enabled (d) )return 0 ;/* Prevent device assign if mem paging or mem sharing have been
@@ -1483,7 +1483,7 @@ int deassign_device(struct domain *d, u16 seg, u8 bus, u8 devfn)struct pci_dev *pdev =NULL ;int ret =0 ; -if ( !iommu_enabled || !hd->platform_ops ) +if ( !is_iommu_enabled (d) )return -EINVAL ;ASSERT (pcidevs_locked ());
@@ -1536,7 +1536,7 @@ static int iommu_get_device_group(int i =0 ;const struct iommu_ops *ops = hd->platform_ops; -if ( !iommu_enabled || !ops || !ops->get_device_group_id ) +if ( !is_iommu_enabled (d) || !ops->get_device_group_id )return 0 ; group_id = ops->get_device_group_id (seg, bus, devfn);
xen/drivers/passthrough/vtd/iommu.c +1 -1
@@ -1757,7 +1757,7 @@ static void iommu_domain_teardown(struct domain *d)xfree (mrmrr); } -ASSERT (iommu_enabled); +ASSERT (is_iommu_enabled (d));/* * We can't use iommu_use_hap_pt here because either IOMMU state
xen/drivers/passthrough/vtd/x86/hvm.c +1 -1
@@ -51,7 +51,7 @@ void hvm_dpci_isairq_eoi(struct domain *d, unsigned int isairq)struct hvm_irq_dpci *dpci =NULL ;ASSERT (isairq <NR_ISAIRQS ); -if ( !iommu_enabled ) +if ( !is_iommu_enabled (d) )return ;spin_lock (&d->event_lock);
xen/drivers/passthrough/x86/iommu.c +1 -1
@@ -178,7 +178,7 @@ int arch_iommu_populate_page_table(struct domain *d) void __hwdom_init arch_iommu_check_autotranslated_hwdom(struct domain *d) { -if ( !iommu_enabled ) +if ( !is_iommu_enabled (d) )panic ("Presently, iommu must be enabled for PVH hardware domain\n" ); }
xen/xsm/flask/hooks.c +9 -9
@@ -886,7 +886,7 @@ static int flask_map_domain_msi (struct domain *d, int irq, const void *data,#endif } -static u32 flask_iommu_resource_use_perm (void ) +static u32 flask_iommu_resource_use_perm (struct domain *d) {/* Obtain the permission level required for allowing a domain * to use an assigned device.
@@ -899,7 +899,7 @@ static u32 flask_iommu_resource_use_perm(void)*/ u32 perm =RESOURCE__USE_NOIOMMU ; -if (iommu_enabled) +if (is_iommu_enabled (d) ) perm = ( iommu_intremap ?RESOURCE__USE_IOMMU :RESOURCE__USE_IOMMU_NOINTREMAP );return perm;
@@ -910,7 +910,7 @@ static int flask_map_domain_irq (struct domain *d, int irq, const void *data)u32 sid, dsid;int rc = -EPERM ;struct avc_audit_data ad; -u32 dperm =flask_iommu_resource_use_perm (); +u32 dperm =flask_iommu_resource_use_perm (d);if ( irq >= nr_static_irqs && data ) { rc =flask_map_domain_msi (d, irq, data, &sid, &ad);
@@ -976,7 +976,7 @@ static int flask_bind_pt_irq (struct domain *d, struct xen_domctl_bind_pt_irq *bint rc = -EPERM ;int irq;struct avc_audit_data ad; -u32 dperm =flask_iommu_resource_use_perm (); +u32 dperm =flask_iommu_resource_use_perm (d); rc =current_has_perm (d,SECCLASS_RESOURCE ,RESOURCE__ADD );if ( rc )
@@ -1049,7 +1049,7 @@ static int flask_iomem_permission(struct domain *d, uint64_t start, uint64_t end data.ssid =domain_sid (current->domain); data.dsid =domain_sid (d); - data.use_perm =flask_iommu_resource_use_perm (); + data.use_perm =flask_iommu_resource_use_perm (d);return security_iterate_iomem_sids (start, end, _iomem_has_perm, &data); }
@@ -1074,7 +1074,7 @@ static int flask_pci_config_permission(struct domain *d, uint32_t machine_bdf, uif ( access && (end >=0x10 && start <0x28 ) ) perm =RESOURCE__SETUP ;else - perm =flask_iommu_resource_use_perm (); + perm =flask_iommu_resource_use_perm (d);AVC_AUDIT_DATA_INIT (&ad,DEV ); ad.device = (unsigned long ) machine_bdf;
@@ -1299,7 +1299,7 @@ static int flask_assign_device(struct domain *d, uint32_t machine_bdf)u32 dsid, rsid;int rc = -EPERM ;struct avc_audit_data ad; -u32 dperm =flask_iommu_resource_use_perm (); +u32 dperm =flask_iommu_resource_use_perm (d);if ( !d )return flask_test_assign_device (machine_bdf);
@@ -1358,7 +1358,7 @@ static int flask_assign_dtdevice(struct domain *d, const char *dtpath)u32 dsid, rsid;int rc = -EPERM ;struct avc_audit_data ad; -u32 dperm =flask_iommu_resource_use_perm (); +u32 dperm =flask_iommu_resource_use_perm (d);if ( !d )return flask_test_assign_dtdevice (dtpath);
@@ -1543,7 +1543,7 @@ static int flask_ioport_permission(struct domain *d, uint32_t start, uint32_t en data.ssid =domain_sid (current->domain); data.dsid =domain_sid (d); - data.use_perm =flask_iommu_resource_use_perm (); + data.use_perm =flask_iommu_resource_use_perm (d);return security_iterate_ioport_sids (start, end, _ioport_has_perm, &data); }
Signature · 7 lines
2.20.1.2.gb21ebb671 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 7/10 remove late (on-demand) construction of IOMMU page tables
In reply to Paul DurrantNow that there is a per-domain IOMMU enable flag, which should be enabled if any device is going to be passed through, stop deferring page table construction until the assignment is done. Also don't tear down the tables again when the last device is de-assigned; defer that task until domain destruction.
This allows the has_iommu_pt() helper and iommu_status enumeration to be removed. Calls to has_iommu_pt() are simply replaced by calls to is_iommu_enabled(). Remaining open-code tests of iommu_hap_pt_share can also be replaced by calls to iommu_use_hap_pt().
The arch_iommu_populate_page_table() and iommu_construct() functions become redundant, as does the 'strict mode' dom0 page_list mapping code in iommu_hwdom_init(), and iommu_teardown() can be made static is its only remaining caller, iommu_domain_destroy(), is within the same source module.
All in all, about 220 lines of code are removed.
NOTE: This patch will cause a small amount of extra resource to be used
to accommodate IOMMU page tables that may never be used, since the per-domain IOMMU flag enable flag is currently set to the value of the global iommu_enable flag. A subsequent patch will add an option to the toolstack to allow it to be turned off if there is no intention to assign passthrough hardware to the domain.
---
Cc: Stefano Stabellini <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Tamas K Lengyel <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Razvan Cojocaru <[email protected]>
Cc: Petre Pircalabu <[email protected]>
Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html
v5:
- Minor style fixes
--- xen/arch/arm/p2m.c | 2 +- xen/arch/x86/dom0_build.c | 2 +- xen/arch/x86/hvm/mtrr.c | 5 +- xen/arch/x86/mm/mem_sharing.c | 2 +- xen/arch/x86/mm/paging.c | 2 +- xen/arch/x86/x86_64/mm.c | 2 +- xen/common/memory.c | 4 +- xen/common/vm_event.c | 2 +- xen/drivers/passthrough/device_tree.c | 11 --- xen/drivers/passthrough/iommu.c | 134 ++++++-------------------- xen/drivers/passthrough/pci.c | 12 --- xen/drivers/passthrough/vtd/iommu.c | 10 +- xen/drivers/passthrough/x86/iommu.c | 95 ------------------ xen/include/asm-arm/iommu.h | 2 +- xen/include/asm-x86/iommu.h | 2 +- xen/include/xen/iommu.h | 16 --- xen/include/xen/sched.h | 2 - 17 files changed, 42 insertions(+), 263 deletions(-)
xen/arch/arm/p2m.c +1 -1
@@ -1056,7 +1056,7 @@ static int __p2m_set_entry(struct p2m_domain *p2m, !mfn_eq(lpae_get_mfn(*entry), lpae_get_mfn(orig_pte)) )p2m_free_entry (p2m, orig_pte, level); -if (has_iommu_pt (p2m->domain) && +if (is_iommu_enabled (p2m->domain) && (lpae_is_valid (orig_pte) ||lpae_is_valid (*entry)) ) {unsigned int flush_flags =0 ;
xen/arch/x86/dom0_build.c +1 -1
@@ -365,7 +365,7 @@ unsigned long __init dom0_compute_nr_pages( } need_paging =is_hvm_domain (d) && - (!iommu_hap_pt_share || !paging_mode_hap (d)); + (!iommu_use_hap_pt (d) || !paging_mode_hap (d));for ( ; ; need_paging = false ) { nr_pages =get_memsize (&dom0_size, avail);
xen/arch/x86/hvm/mtrr.c +3 -2
@@ -783,7 +783,8 @@ HVM_REGISTER_SAVE_RESTORE(MTRR, hvm_save_mtrr_msr, hvm_load_mtrr_msr, 1,void memory_type_changed (struct domain *d) { -if ( (has_iommu_pt (d) ||cache_flush_permitted (d)) && d->vcpu && d->vcpu[0 ] ) +if ( (is_iommu_enabled (d) ||cache_flush_permitted (d)) && + d->vcpu && d->vcpu[0 ] ) {p2m_memory_type_changed (d);flush_all (FLUSH_CACHE );
@@ -831,7 +832,7 @@ int epte_get_entry_emt(struct domain *d, unsigned long gfn, mfn_t mfn,return MTRR_TYPE_UNCACHABLE ; } -if ( !has_iommu_pt (d) && !cache_flush_permitted (d) ) +if ( !is_iommu_enabled (d) && !cache_flush_permitted (d) ) { *ipat =1 ;return MTRR_TYPE_WRBACK ;
xen/arch/x86/mm/mem_sharing.c +1 -1
@@ -1664,7 +1664,7 @@ int mem_sharing_domctl(struct domain *d, struct xen_domctl_mem_sharing_op *mec)case XEN_DOMCTL_MEM_SHARING_CONTROL : { rc =0 ; -if (unlikely (has_iommu_pt (d) && mec->u.enable) ) +if (unlikely (is_iommu_enabled (d) && mec->u.enable) ) rc = -EXDEV ;else d->arch.hvm.mem_sharing_enabled = mec->u.enable;
xen/arch/x86/mm/paging.c +1 -1
@@ -213,7 +213,7 @@ int paging_log_dirty_enable(struct domain *d, bool_t log_global) {int ret; -if (has_iommu_pt (d) && log_global ) +if (is_iommu_enabled (d) && log_global ) {/* * Refuse to turn on global log-dirty mode
xen/arch/x86/x86_64/mm.c +1 -1
@@ -1433,7 +1433,7 @@ int memory_add(unsigned long spfn, unsigned long epfn, unsigned int pxm)* shared or being kept in sync then newly added memory needs to be * mapped here. */ -if (has_iommu_pt (hardware_domain) && +if (is_iommu_enabled (hardware_domain) && !iommu_use_hap_pt (hardware_domain) && !need_iommu_pt_sync (hardware_domain) ) {
xen/common/memory.c +2 -2
@@ -823,7 +823,7 @@ int xenmem_add_to_physmap(struct domain *d, struct xen_add_to_physmap *xatp, xatp->gpfn += start; xatp->size -= start; -if (has_iommu_pt (d) ) +if (is_iommu_enabled (d) )this_cpu (iommu_dont_flush_iotlb) =1 ;while ( xatp->size > done )
@@ -844,7 +844,7 @@ int xenmem_add_to_physmap(struct domain *d, struct xen_add_to_physmap *xatp, } } -if (has_iommu_pt (d) ) +if (is_iommu_enabled (d) ) {int ret;
xen/common/vm_event.c +1 -1
@@ -630,7 +630,7 @@ int vm_event_domctl(struct domain *d, struct xen_domctl_vm_event_op *vec)/* No paging if iommu is used */ rc = -EMLINK ; -if (unlikely (has_iommu_pt (d)) ) +if (unlikely (is_iommu_enabled (d)) )break ; rc = -EXDEV ;
xen/drivers/passthrough/device_tree.c +0 -11
@@ -40,17 +40,6 @@ int iommu_assign_dt_device(struct domain *d, struct dt_device_node *dev)if ( !list_empty (&dev->domain_list) ) goto fail; -/* -* The hwdom is forced to use IOMMU for protecting assigned -* device. Therefore the IOMMU data is already set up. -*/ -ASSERT (!is_hardware_domain (d) || - hd->status == IOMMU_STATUS_initialized); - - rc =iommu_construct (d); -if ( rc ) - goto fail; -/* The flag field doesn't matter to DT device. */ rc = hd->platform_ops->assign_device (d,0 ,dt_to_dev (dev),0 );
xen/drivers/passthrough/iommu.c +28 -106
182 lines not on this page · raw message
xen/drivers/passthrough/pci.c +0 -12
@@ -1434,13 +1434,6 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag)if ( !pcidevs_trylock () )return -ERESTART ; - rc =iommu_construct (d); -if ( rc ) - { -pcidevs_unlock (); -return rc; - } - pdev =pci_get_pdev_by_domain (hardware_domain, seg, bus, devfn);if ( !pdev ) {
@@ -1469,8 +1462,6 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag) } done: -if ( !has_arch_pdevs (d) &&has_iommu_pt (d) ) -iommu_teardown (d);pcidevs_unlock ();return rc;
@@ -1519,9 +1510,6 @@ int deassign_device(struct domain *d, u16 seg, u8 bus, u8 devfn) pdev->fault.count =0 ; -if ( !has_arch_pdevs (d) &&has_iommu_pt (d) ) -iommu_teardown (d); -return ret; }
xen/drivers/passthrough/vtd/iommu.c +1 -9
@@ -1759,15 +1759,7 @@ static void iommu_domain_teardown(struct domain *d)ASSERT (is_iommu_enabled (d)); -/* -* We can't use iommu_use_hap_pt here because either IOMMU state -* is already changed to IOMMU_STATUS_disabled at this point or -* has always been IOMMU_STATUS_disabled. -* -* We also need to test if HAP is enabled because PV guests can -* enter this path too. -*/ -if (hap_enabled (d) && iommu_hap_pt_share ) +if (iommu_use_hap_pt (d) )return ;spin_lock (&hd->arch.mapping_lock);
xen/drivers/passthrough/x86/iommu.c +0 -95
101 lines not on this page · raw message
xen/include/asm-arm/iommu.h +1 -1
@@ -21,7 +21,7 @@ struct arch_iommu };/* Always share P2M Table between the CPU and the IOMMU */ -#define iommu_use_hap_pt (d) (has_iommu_pt(d)) +#define iommu_use_hap_pt (d) is_iommu_enabled(d)const struct iommu_ops *iommu_get_ops (void );void iommu_set_ops (const struct iommu_ops *ops);
xen/include/asm-x86/iommu.h +1 -1
@@ -81,7 +81,7 @@ extern const struct iommu_init_ops *iommu_init_ops;/* Are we using the domain P2M table as its IOMMU pagetable? */ #define iommu_use_hap_pt (d) \ - (hap_enabled(d) && has_iommu_pt(d) && iommu_hap_pt_share) + (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share)void iommu_update_ire_from_apic (unsigned int apic,unsigned int reg,unsigned int value);unsigned int iommu_read_apic_from_ire (unsigned int apic,unsigned int reg);
xen/include/xen/iommu.h +0 -16
@@ -74,15 +74,9 @@ int deassign_device(struct domain *d, u16 seg, u8 bus, u8 devfn);void arch_iommu_domain_destroy (struct domain *d);int arch_iommu_domain_init (struct domain *d); -int arch_iommu_populate_page_table (struct domain *d);void arch_iommu_check_autotranslated_hwdom (struct domain *d);void arch_iommu_hwdom_init (struct domain *d); -int iommu_construct (struct domain *d); - -/* Function used internally, use iommu_domain_destroy */ -void iommu_teardown (struct domain *d); -/* * The following flags are passed to map operations and passed by lookup * operations.
@@ -249,13 +243,6 @@ struct iommu_ops {# define iommu_vcall iommu_call#endif -enum iommu_status -{ - IOMMU_STATUS_disabled, - IOMMU_STATUS_initializing, - IOMMU_STATUS_initialized -}; -struct domain_iommu {struct arch_iommu arch;
@@ -270,9 +257,6 @@ struct domain_iommu {/* Features supported by the IOMMU */ DECLARE_BITMAP (features, IOMMU_FEAT_count); -/* Status of guest IOMMU mappings */ -enum iommu_status status; -/* * Does the guest reqire mappings to be synchonized, to maintain * the default dfn == pfn map. (See comment on dfn at the top of
xen/include/xen/sched.h +0 -2
@@ -964,10 +964,8 @@ static inline bool is_hwdom_pinned_vcpu(const struct vcpu *v) }#ifdef CONFIG_HAS_PASSTHROUGH -#define has_iommu_pt (d) (dom_iommu(d)->status != IOMMU_STATUS_disabled)#define need_iommu_pt_sync (d) (dom_iommu(d)->need_sync)#else -#define has_iommu_pt (d) false#define need_iommu_pt_sync (d) false#endif
Signature · 7 lines
2.20.1.2.gb21ebb671 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 8/10 make passthrough/pci.c:deassign_device() static
In reply to Paul DurrantThis function is only ever called from within the same source module and really has no business being declared xen/iommu.h. This patch relocates the function ahead of the first called and makes it static.
---
Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html
v5:
- minor style fixes
- use %pd, rather than d%d
--- xen/drivers/passthrough/pci.c | 93 ++++++++++++++++++----------------- xen/include/xen/iommu.h | 1 - 2 files changed, 47 insertions(+), 47 deletions(-)
xen/drivers/passthrough/pci.c +47 -46
@@ -889,6 +889,53 @@ static int pci_clean_dpci_irqs(struct domain *d)return 0 ; } +/* caller should hold the pcidevs_lock */ +static int deassign_device (struct domain *d,uint16_t seg,uint8_t bus, +uint8_t devfn) +{ +const struct domain_iommu *hd =dom_iommu (d); +struct pci_dev *pdev; +int ret =0 ; + +if ( !is_iommu_enabled (d) ) +return -EINVAL ; + +ASSERT (pcidevs_locked ()); + pdev =pci_get_pdev_by_domain (d, seg, bus, devfn); +if ( !pdev ) +return -ENODEV ; + +while ( pdev->phantom_stride ) + { + devfn += pdev->phantom_stride; +if (PCI_SLOT (devfn) !=PCI_SLOT (pdev->devfn) ) +break ; + ret = hd->platform_ops->reassign_device (d, hardware_domain, devfn, +pci_to_dev (pdev)); +if ( !ret ) +continue ; + +printk (XENLOG_G_ERR "%pd: deassign %04x:%02x:%02x.%u failed (%d)\n" , + d, seg, bus,PCI_SLOT (devfn),PCI_FUNC (devfn), ret); +return ret; + } + + devfn = pdev->devfn; + ret = hd->platform_ops->reassign_device (d, hardware_domain, devfn, +pci_to_dev (pdev)); +if ( ret ) + { +dprintk (XENLOG_G_ERR , +"%pd: deassign device (%04x:%02x:%02x.%u) failed\n" , + d, seg, bus,PCI_SLOT (devfn),PCI_FUNC (devfn)); +return ret; + } + + pdev->fault.count =0 ; + +return ret; +} +int pci_release_devices (struct domain *d) {struct pci_dev *pdev;
@@ -1467,52 +1514,6 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag)return rc; } -/* caller should hold the pcidevs_lock */ -int deassign_device (struct domain *d,u16 seg,u8 bus,u8 devfn) -{ -const struct domain_iommu *hd =dom_iommu (d); -struct pci_dev *pdev =NULL ; -int ret =0 ; - -if ( !is_iommu_enabled (d) ) -return -EINVAL ; - -ASSERT (pcidevs_locked ()); - pdev =pci_get_pdev_by_domain (d, seg, bus, devfn); -if ( !pdev ) -return -ENODEV ; - -while ( pdev->phantom_stride ) - { - devfn += pdev->phantom_stride; -if (PCI_SLOT (devfn) !=PCI_SLOT (pdev->devfn) ) -break ; - ret = hd->platform_ops->reassign_device (d, hardware_domain, devfn, -pci_to_dev (pdev)); -if ( !ret ) -continue ; - -printk (XENLOG_G_ERR "d%d: deassign %04x:%02x:%02x.%u failed (%d)\n" , - d->domain_id, seg, bus,PCI_SLOT (devfn),PCI_FUNC (devfn), ret); -return ret; - } - - devfn = pdev->devfn; - ret = hd->platform_ops->reassign_device (d, hardware_domain, devfn, -pci_to_dev (pdev)); -if ( ret ) - { -dprintk (XENLOG_G_ERR , -"d%d: deassign device (%04x:%02x:%02x.%u) failed\n" , - d->domain_id, seg, bus,PCI_SLOT (devfn),PCI_FUNC (devfn)); -return ret; - } - - pdev->fault.count =0 ; - -return ret; -} -static int iommu_get_device_group (struct domain *d,u16 seg,u8 bus,u8 devfn,XEN_GUEST_HANDLE_64 (uint32) buf,int max_sdevs)
xen/include/xen/iommu.h +0 -1
@@ -70,7 +70,6 @@ int iommu_hardware_setup(void);int iommu_domain_init (struct domain *d);void iommu_hwdom_init (struct domain *d);void iommu_domain_destroy (struct domain *d); -int deassign_device (struct domain *d,u16 seg,u8 bus,u8 devfn);void arch_iommu_domain_destroy (struct domain *d);int arch_iommu_domain_init (struct domain *d);
Signature · 7 lines
2.20.1.2.gb21ebb671 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 9/10 iommu: tidy up iommu_use_hap_pt() and need_iommu_pt_sync() macros
In reply to Paul DurrantThes macros really ought to live in the common xen/iommu.h header rather then being distributed amongst architecture specific iommu headers and xen/sched.h. This patch moves them there.
NOTE: Disabling 'sharept' in the command line iommu options should really
be hard error on ARM (as opposed to just being ignored), so avoid parsing that option if CONFIG_ARM is set.
---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Previously part of https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html
--- xen/drivers/passthrough/iommu.c | 2 ++ xen/include/asm-arm/iommu.h | 3 --- xen/include/asm-x86/iommu.h | 4 ---- xen/include/xen/iommu.h | 11 +++++++++++ xen/include/xen/sched.h | 6 ------ 5 files changed, 13 insertions(+), 13 deletions(-)
xen/drivers/passthrough/iommu.c +2 -0
@@ -102,8 +102,10 @@ static int __init parse_iommu_param(const char *s) iommu_hwdom_passthrough = val; else if ( (val = parse_boolean("dom0-strict", s, ss)) >= 0 ) iommu_hwdom_strict = val; +#ifndef CONFIG_ARM else if ( (val = parse_boolean("sharept", s, ss)) >= 0 ) iommu_hap_pt_share = val; +#endif else rc = -EINVAL ;
xen/include/asm-arm/iommu.h +0 -3
@@ -20,9 +20,6 @@ struct arch_iommuvoid *priv; }; -/* Always share P2M Table between the CPU and the IOMMU */ -#define iommu_use_hap_pt (d) is_iommu_enabled(d) -const struct iommu_ops *iommu_get_ops (void );void iommu_set_ops (const struct iommu_ops *ops);
xen/include/asm-x86/iommu.h +0 -4
@@ -79,10 +79,6 @@ struct iommu_init_ops {extern const struct iommu_init_ops *iommu_init_ops; -/* Are we using the domain P2M table as its IOMMU pagetable? */ -#define iommu_use_hap_pt (d) \ - (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share) -void iommu_update_ire_from_apic (unsigned int apic,unsigned int reg,unsigned int value);unsigned int iommu_read_apic_from_ire (unsigned int apic,unsigned int reg);int iommu_setup_hpet_msi (struct msi_desc *);
xen/include/xen/iommu.h +11 -0
@@ -268,6 +268,17 @@ struct domain_iommu {#define iommu_set_feature (d, f) set_bit(f, dom_iommu(d)->features)#define iommu_clear_feature (d, f) clear_bit(f, dom_iommu(d)->features) +/* Are we using the domain P2M table as its IOMMU pagetable? */ +#define iommu_use_hap_pt (d) \ + (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share) + +/* Does the IOMMU pagetable need to be kept synchronized with the P2M */ +#ifdef CONFIG_HAS_PASSTHROUGH +#define need_iommu_pt_sync (d) (dom_iommu(d)->need_sync) +#else +#define need_iommu_pt_sync (d) false +#endif +int __must_check iommu_suspend (void );void iommu_resume (void );void iommu_crash_shutdown (void );
xen/include/xen/sched.h +0 -6
@@ -963,12 +963,6 @@ static inline bool is_hwdom_pinned_vcpu(const struct vcpu *v) cpumask_weight(v->cpu_hard_affinity) == 1); } -#ifdef CONFIG_HAS_PASSTHROUGH -#define need_iommu_pt_sync (d) (dom_iommu(d)->need_sync) -#else -#define need_iommu_pt_sync (d) false -#endif -static inline bool is_vcpu_online (const struct vcpu *v) {return !test_bit (_VPF_down, &v->pause_flags);
Signature · 7 lines
2.20.1.2.gb21ebb671 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 10/10 introduce a 'passthrough' configuration option to xl.cfg...
In reply to Paul Durrant...and hence the ability to disable IOMMU mappings, and control EPT sharing.
This patch introduces a new 'libxl_passthrough' enumeration into libxl_domain_create_info. The value will be set by xl either when it parses a new 'passthrough' option in xl.cfg, or implicitly if there is passthrough hardware specified for the domain.
If the value of the passthrough configuration option is 'disabled' then the XEN_DOMCTL_CDF_iommu flag will be clear in the xen_domctl_createdomain flags, thus allowing the toolstack to control whether the domain gets IOMMU mappings or not (where previously they were globally set).
If the value of the passthrough configuration option is 'sync_pt' then a new 'iommu_opts' field in xen_domctl_createdomain will be set with the value XEN_DOMCTL_IOMMU_no_sharept. This will override the global default set in iommu_hap_pt_share, thus allowing the toolstack to control whether EPT sharing is used for the domain.
NOTE: The call to libxl_get_physinfo() in libxl__domain_make() is left
in place to allow attempts to passthrough hardware on a hypervisor with disabled IOMMU support to be rejected early.
---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html
v5:
- Expand xen_domctl_createdomain flags field and hence bump interface version
- Fix spelling mistakes in context line
--- docs/man/xl.cfg.5.pod.in | 52 +++++++++++++++++++++++++++++++++ tools/libxl/libxl.h | 5 ++++ tools/libxl/libxl_create.c | 16 +++++++++- tools/libxl/libxl_types.idl | 7 +++++ tools/xl/xl_parse.c | 38 ++++++++++++++++++++++++ xen/arch/arm/domain.c | 10 ++++++- xen/arch/x86/domain.c | 2 +- xen/common/domain.c | 7 +++++ xen/drivers/passthrough/iommu.c | 13 ++++++++- xen/include/public/domctl.h | 6 +++- xen/include/xen/iommu.h | 19 ++++++++---- 11 files changed, 164 insertions(+), 11 deletions(-)
docs/man/xl.cfg.5.pod.in +52 -0
@@ -605,6 +605,58 @@ option should only be used with a trusted device tree. Note that the partial device tree should avoid using the phandle 65000 which is reserved by the toolstack. +=item B<passthrough="STRING"> + +Specify whether IOMMU mappings are enabled for the domain and hence whether +it will be enabled for passthrough hardware. Valid values for this option +are: + +=over 4 + +=item B<disabled> + +IOMMU mappings are disabled for the domain and so hardware may not be +passed through. + +This option is the default if no passthrough hardware is specified +in the domain's configuration. + +=item B<sync_pt> + +This option means that IOMMU mappings will be synchronized with the +domain's P2M table as follows: + +For a PV domain, all writable pages assigned to the domain are identity +mapped by MFN in the IOMMU page table. Thus a device driver running in the +domain may program passthrough hardware for DMA using MFN values +(i.e. host/machine frame numbers) looked up in its P2M. + +For an HVM domain, all non-foreign RAM pages present in its P2M will be +mapped by GFN in the IOMMU page table. Thus a device driver running in the +domain may program passthrough hardware using GFN values (i.e. guest +physical frame numbers) without any further translation. + +This option is the default if the domain is PV and passthrough hardware +is specified in the configuration. + +This option is not available on Arm. + +=item B<share_pt> + +This option is unavailable for a PV domain. For an HVM domain, this option +means that the IOMMU will be programmed to directly reference the domain's +P2M table as its page table. From the point of view of a device driver +running in the domain this is functionally equivalent to B<sync_pt> but +places less load on the hypervisor and so should generally be selected in +preference. However, the availability of this option is hardware specific +and thus, if it is specified for a domain running on hardware that does +not allow it, B<sync_pt> will be used instead. + +This option is the default if the domain is HVM and passthrough hardware +is specified in the configuration. + +=back + =back =head2 Devices
tools/libxl/libxl.h +5 -0
@@ -394,6 +394,11 @@*/ #define LIBXL_HAVE_EXTENDED_VKB 1 +/* +* libxl_domain_create_info has libxl_passthrough enumeration. +*/ +#define LIBXL_HAVE_CREATEINFO_PASSTHROUGH 1 +/* * libxl ABI compatibility *
tools/libxl/libxl_create.c +15 -1
@@ -565,12 +565,26 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,libxl_defbool_val (info->oos) ?0 : XEN_DOMCTL_CDF_oos_off; } +LOG (DETAIL ,"passthrough: %s" , +libxl_passthrough_to_string (info->passthrough)); + rc =libxl_get_physinfo (ctx, &physinfo);if (rc <0 ) goto out; -if (physinfo.cap_hvm_directio) +if (info->passthrough !=LIBXL_PASSTHROUGH_DISABLED ) + { +if (!physinfo.cap_hvm_directio) { +LOGED (ERROR , *domid,"passthrough not available" ); + rc =ERROR_FAIL ; + goto out; + } + create.flags |= XEN_DOMCTL_CDF_iommu; + } + +if (info->passthrough ==LIBXL_PASSTHROUGH_SYNC_PT ) + create.iommu_opts |= XEN_DOMCTL_IOMMU_no_sharept;/* Ultimately, handle is an array of 16 uint8_t, same as uuid */ libxl_uuid_copy (ctx, (libxl_uuid *)&create.handle, &info->uuid);
tools/libxl/libxl_types.idl +7 -0
@@ -263,6 +263,12 @@ libxl_vkb_backend = Enumeration("vkb_backend", [ (2, "LINUX") ]) +libxl_passthrough = Enumeration("passthrough", [ + (0, "disabled"), + (1, "sync_pt"), + (2, "share_pt"), + ]) + # # Complex libxl types #
@@ -408,6 +414,7 @@ libxl_domain_create_info = Struct("domain_create_info",[ ("pool_name", string), ("run_hotplug_scripts",libxl_defbool), ("driver_domain",libxl_defbool), + ("passthrough", libxl_passthrough), ], dir=DIR_IN) libxl_domain_restore_params = Struct("domain_restore_params", [
tools/xl/xl_parse.c +38 -0
@@ -2326,6 +2326,44 @@ skip_vfb: } } +if (!xlu_cfg_get_string (config,"passthrough" , &buf,0 )) { +libxl_passthrough o; + + e =libxl_passthrough_from_string (buf, &o); +if (e) { +fprintf (stderr, +"ERROR: unknown passthrough option '%s'\n" , + buf); +exit (-ERROR_FAIL ); + } + +switch (o) { +case LIBXL_PASSTHROUGH_DISABLED : +if (d_config->num_pcidevs || d_config->num_dtdevs) { +fprintf (stderr, +"ERROR: passthrough disabled but devices are specified\n" ); +exit (-ERROR_FAIL ); + } +case LIBXL_PASSTHROUGH_SHARE_PT : +if (c_info->type ==LIBXL_DOMAIN_TYPE_PV ) { +fprintf (stderr, +"ERROR: passthrough=\"share_pt\" not valid for PV domain\n" ); +exit (-ERROR_FAIL ); + } +default : +break ; + } + + c_info->passthrough = o; + }else if (d_config->num_pcidevs || d_config->num_dtdevs) { +/* +* Passthrough devices are specified so set an appropriate +* default value. +*/ + c_info->passthrough = (c_info->type ==LIBXL_DOMAIN_TYPE_PV ) ? +LIBXL_PASSTHROUGH_SYNC_PT :LIBXL_PASSTHROUGH_SHARE_PT ; + } +if (!xlu_cfg_get_list (config,"usbctrl" , &usbctrls,0 ,0 )) { d_config->num_usbctrls =0 ; d_config->usbctrls =NULL ;
xen/arch/arm/domain.c +9 -1
@@ -614,6 +614,14 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +/* Always share P2M Table between the CPU and the IOMMU */ +if ( config->iommu_opts & XEN_DOMCTL_IOMMU_no_sharept ) + { +dprintk (XENLOG_INFO , +"Unsupported iommu option: XEN_DOMCTL_IOMMU_no_sharept\n" ); +return -EINVAL ; + } +/* Fill in the native GIC version, passed back to the toolstack. */ if ( config->arch.gic_version ==XEN_DOMCTL_CONFIG_GIC_NATIVE ) {
@@ -673,7 +681,7 @@ int arch_domain_create(struct domain *d,ASSERT (config !=NULL ); -if ( (rc =iommu_domain_init (d)) !=0 ) +if ( (rc =iommu_domain_init (d, config->iommu_opts)) !=0 ) goto fail;if ( (rc =p2m_init (d)) !=0 )
xen/arch/x86/domain.c +1 -1
@@ -604,7 +604,7 @@ int arch_domain_create(struct domain *d,if ( (rc =init_domain_irq_mapping (d)) !=0 ) goto fail; -if ( (rc =iommu_domain_init (d)) !=0 ) +if ( (rc =iommu_domain_init (d, config->iommu_opts)) !=0 ) goto fail;psr_domain_init (d);
xen/common/domain.c +7 -0
@@ -308,6 +308,13 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)return -EINVAL ; } +if ( !(config->flags & XEN_DOMCTL_CDF_iommu) && config->iommu_opts ) + { +dprintk (XENLOG_INFO , +"IOMMU options specified but IOMMU not enabled\n" ); +return -EINVAL ; + } +if ( config->max_vcpus <1 ) {dprintk (XENLOG_INFO ,"No vCPUS\n" );
xen/drivers/passthrough/iommu.c +12 -1
@@ -159,7 +159,7 @@ static void __hwdom_init check_hwdom_reqs(struct domain *d) iommu_hwdom_strict = true; } -int iommu_domain_init (struct domain *d) +int iommu_domain_init (struct domain *d,unsigned int opts) {struct domain_iommu *hd =dom_iommu (d);int ret =0 ;
@@ -176,6 +176,15 @@ int iommu_domain_init(struct domain *d)if ( ret )return ret; +/* +* Use shared page tables for HAP and IOMMU if the global option +* is enabled (from which we can infer the h/w is capable) and +* the domain options do not disallow it. HAP must, of course, also +* be enabled. +*/ + hd->hap_pt_share =hap_enabled (d) && iommu_hap_pt_share && + !(opts & XEN_DOMCTL_IOMMU_no_sharept); +/* * NB: 'relaxed' h/w domains don't need the IOMMU mappings to be kept * in-sync with their assigned pages because all host RAM will be
@@ -187,6 +196,8 @@ int iommu_domain_init(struct domain *d)if ( !is_hardware_domain (d) || iommu_hwdom_strict ) hd->need_sync = !iommu_use_hap_pt (d); +ASSERT (!(hd->need_sync && hd->hap_pt_share)); +return 0 ; }
xen/include/public/domctl.h +5 -1
@@ -38,7 +38,7 @@#include "hvm/save.h" #include "memory.h" -#define XEN_DOMCTL_INTERFACE_VERSION 0x00000011 +#define XEN_DOMCTL_INTERFACE_VERSION 0x00000012/* * NB. xen_domctl.domain is an IN/OUT parameter for this operation.
@@ -70,6 +70,10 @@ struct xen_domctl_createdomain {uint32_t flags; +#define _XEN_DOMCTL_IOMMU_no_sharept 0 +#define XEN_DOMCTL_IOMMU_no_sharept (1U<<_XEN_DOMCTL_IOMMU_no_sharept) +uint32_t iommu_opts; +/* * Various domain limits, which impact the quantity of resources (global * mapping space, xenheap, etc) a guest may consume.
xen/include/xen/iommu.h +13 -6
@@ -67,7 +67,7 @@ extern unsigned int iommu_dev_iotlb_timeout;int iommu_setup (void );int iommu_hardware_setup (void ); -int iommu_domain_init (struct domain *d); +int iommu_domain_init (struct domain *d,unsigned int opts);void iommu_hwdom_init (struct domain *d);void iommu_domain_destroy (struct domain *d);
@@ -257,9 +257,17 @@ struct domain_iommu {DECLARE_BITMAP (features, IOMMU_FEAT_count);/* -* Does the guest reqire mappings to be synchonized, to maintain -* the default dfn == pfn map. (See comment on dfn at the top of -* include/xen/mm.h). +* Does the guest share HAP mapping with the IOMMU? This is always +* true for ARM systems and may be true for x86 systems where the +* the hardware is capable. +*/ +bool hap_pt_share; + +/* +* Does the guest require mappings to be synchronized, to maintain +* the default dfn == pfn map? (See comment on dfn at the top of +* include/xen/mm.h). Note that hap_pt_share == false does not +* necessarily imply this is true. */ bool need_sync; };
@@ -269,8 +277,7 @@ struct domain_iommu {#define iommu_clear_feature (d, f) clear_bit(f, dom_iommu(d)->features)/* Are we using the domain P2M table as its IOMMU pagetable? */ -#define iommu_use_hap_pt (d) \ - (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share) +#define iommu_use_hap_pt (d) (dom_iommu(d)->hap_pt_share)/* Does the IOMMU pagetable need to be kept synchronized with the P2M */ #ifdef CONFIG_HAS_PASSTHROUGH
Signature · 7 lines
2.20.1.2.gb21ebb671 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel