Skip to contentSkip to conversation

add per-domain IOMMU control

PATCH Superseded Paul Durrant 5 participants 29 messages Sep 2, 2019 – Sep 6, 2019

v8 of 13 Superseded Latest is v13 →

lore
Share & tools
lore
#0 lore raw

v8 cover add per-domain IOMMU control

These are revisions of the remaining uncommitted patches from my previous series:

https://lists.xenproject.org/archives/html/xen-devel/2019-08/msg01737.html

Paul Durrant (6):
x86/domain: remove the 'oos_off' flag
domain: introduce XEN_DOMCTL_CDF_iommu flag
use is_iommu_enabled() where appropriate...
remove late (on-demand) construction of IOMMU page tables
iommu: tidy up iommu_use_hap_pt() and need_iommu_pt_sync() macros introduce a 'passthrough' configuration option to xl.cfg...

 docs/man/xl.cfg.5.pod.in                  |  52 +++++++
 tools/libxl/libxl.h                       |   5 +
 tools/libxl/libxl_create.c                |   9 ++
 tools/libxl/libxl_mem.c                   |   6 +-
 tools/libxl/libxl_types.idl               |   8 +
 tools/libxl/libxl_utils.c                 |  15 ++
 tools/libxl/libxl_utils.h                 |   1 +
 tools/ocaml/libs/xc/xenctrl.ml            |  12 +-
 tools/ocaml/libs/xc/xenctrl.mli           |  12 +-
 tools/ocaml/libs/xc/xenctrl_stubs.c       |  15 +-
 tools/xl/xl_parse.c                       | 145 +++++++++++------
 xen/arch/arm/domain.c                     |  10 +-
 xen/arch/arm/p2m.c                        |   4 +-
 xen/arch/arm/setup.c                      |   3 +
 xen/arch/x86/dom0_build.c                 |   4 +-
 xen/arch/x86/domain.c                     |   9 +-
 xen/arch/x86/domctl.c                     |   4 +-
 xen/arch/x86/hvm/hvm.c                    |   6 +-
 xen/arch/x86/hvm/mtrr.c                   |   5 +-
 xen/arch/x86/hvm/vioapic.c                |   2 +-
 xen/arch/x86/hvm/vmx/vmcs.c               |   2 +-
 xen/arch/x86/hvm/vmx/vmx.c                |   2 +-
 xen/arch/x86/mm/mem_sharing.c             |   2 +-
 xen/arch/x86/mm/p2m-ept.c                 |   4 +-
 xen/arch/x86/mm/p2m.c                     |   4 +-
 xen/arch/x86/mm/paging.c                  |   4 +-
 xen/arch/x86/mm/shadow/common.c           |   7 +-
 xen/arch/x86/mm/shadow/none.c             |   2 +-
 xen/arch/x86/setup.c                      |   3 +
 xen/arch/x86/x86_64/mm.c                  |   2 +-
 xen/common/domain.c                       |  16 +-
 xen/common/memory.c                       |   4 +-
 xen/common/vm_event.c                     |   2 +-
 xen/drivers/passthrough/amd/iommu_guest.c |   2 +-
 xen/drivers/passthrough/device_tree.c     |  18 +--
 xen/drivers/passthrough/io.c              |   8 +-
 xen/drivers/passthrough/iommu.c           | 180 +++++++---------------
 xen/drivers/passthrough/pci.c             |  28 +---
 xen/drivers/passthrough/vtd/iommu.c       |  12 +-
 xen/drivers/passthrough/vtd/x86/hvm.c     |   2 +-
 xen/drivers/passthrough/x86/iommu.c       |  99 +-----------
 xen/include/asm-arm/iommu.h               |   3 -
 xen/include/asm-x86/domain.h              |   1 -
 xen/include/asm-x86/iommu.h               |  17 +-
 xen/include/asm-x86/shadow.h              |   2 +-
 xen/include/public/domctl.h               |  10 +-
 xen/include/xen/iommu.h                   |  48 +++---
 xen/include/xen/sched.h                   |  13 +-
 xen/xsm/flask/hooks.c                     |  18 +--
 49 files changed, 433 insertions(+), 409 deletions(-)

---
Cc: Andrew Cooper <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Christian Lindig <[email protected]>
Cc: David Scott <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Jun Nakajima <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Petre Pircalabu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Suravee Suthikulpanit <[email protected]>
Cc: Tamas K Lengyel <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Wei Liu <[email protected]>

Signature · 7 lines
2.20.1.2.gb21ebb671


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH v8 1/6 x86/domain: remove the 'oos_off' flag

The flag is not needed since the domain 'options' can now be tested directly.

---
Cc: Jan Beulich <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

v8:
- Move setting CDF_oos_off into x86 arch_sanitise_domain_config()
- Dropped Jan's R-b because of the change

v3:
- Force 'oos_off' to be set for PV guests (to avoid call to is_hvm_domain() except in ASSERT)
- Dropped Tim's A-b because of the change

v2:
- Move some of the hunks from patch #3
- Also update the definition of shadow_domain_init() in none.c

---
 xen/arch/x86/domain.c           | 7 +++++++
 xen/arch/x86/mm/paging.c        | 2 +-
 xen/arch/x86/mm/shadow/common.c | 7 ++++---
 xen/arch/x86/mm/shadow/none.c   | 2 +-
 xen/include/asm-x86/domain.h    | 1 -
 xen/include/asm-x86/shadow.h    | 2 +-
 6 files changed, 14 insertions(+), 7 deletions(-)
xen/arch/x86/domain.c +7 -0
index 2df312332d..d5a19404a6 100644
@@ -473,6 +473,13 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( !(config->flags & XEN_DOMCTL_CDF_hvm_guest) )
+        /*
+         * It is only meaningful for XEN_DOMCTL_CDF_oos_off to be clear
+         * for HVM guests.
+         */
+        config->flags |= XEN_DOMCTL_CDF_oos_off;
+
     return 0;
 }
 
xen/arch/x86/mm/paging.c +1 -1
index 097a27f608..69aa228e46 100644
@@ -653,7 +653,7 @@ int paging_domain_init(struct domain *d)
     if ( hap_enabled(d) )
         hap_domain_init(d);
     else
-        rc = shadow_domain_init(d, d->options);
+        rc = shadow_domain_init(d);
 
     return rc;
 }
xen/arch/x86/mm/shadow/common.c +4 -3
index c0d4a27287..9463794059 100644
@@ -46,7 +46,7 @@ static void sh_clean_dirty_bitmap(struct domain *);
 
 /* Set up the shadow-specific parts of a domain struct at start of day.
  * Called for every domain from arch_domain_create() */
-int shadow_domain_init(struct domain *d, unsigned int domcr_flags)
+int shadow_domain_init(struct domain *d)
 {
     static const struct log_dirty_ops sh_ops = {
         .enable  = sh_enable_log_dirty,
@@ -62,7 +62,6 @@ int shadow_domain_init(struct domain *d, unsigned int domcr_flags)
 
 #if (SHADOW_OPTIMIZATIONS & SHOPT_OUT_OF_SYNC)
     d->arch.paging.shadow.oos_active = 0;
-    d->arch.paging.shadow.oos_off = domcr_flags & XEN_DOMCTL_CDF_oos_off;
 #endif
     d->arch.paging.shadow.pagetable_dying_op = 0;
 
@@ -2528,11 +2527,13 @@ static void sh_update_paging_modes(struct vcpu *v)
 #if (SHADOW_OPTIMIZATIONS & SHOPT_OUT_OF_SYNC)
     /* We need to check that all the vcpus have paging enabled to
      * unsync PTs. */
-    if ( is_hvm_domain(d) && !d->arch.paging.shadow.oos_off )
+    if ( !(d->options & XEN_DOMCTL_CDF_oos_off) )
     {
         int pe = 1;
         struct vcpu *vptr;
 
+        ASSERT(is_hvm_domain(d));
+
         for_each_vcpu(d, vptr)
         {
             if ( !hvm_paging_enabled(vptr) )
xen/arch/x86/mm/shadow/none.c +1 -1
index a70888bd98..2fddf4274c 100644
@@ -18,7 +18,7 @@ static void _clean_dirty_bitmap(struct domain *d)
     ASSERT(is_pv_domain(d));
 }
 
-int shadow_domain_init(struct domain *d, unsigned int domcr_flags)
+int shadow_domain_init(struct domain *d)
 {
     static const struct log_dirty_ops sh_none_ops = {
         .enable  = _enable_log_dirty,
xen/include/asm-x86/domain.h +0 -1
index 9f3afd12bc..7cebfa4fb9 100644
@@ -115,7 +115,6 @@ struct shadow_domain {
 
     /* OOS */
     bool_t oos_active;
-    bool_t oos_off;
 
     /* Has this domain ever used HVMOP_pagetable_dying? */
     bool_t pagetable_dying_op;
xen/include/asm-x86/shadow.h +1 -1
index f29f0f652b..8ebb89c027 100644
@@ -49,7 +49,7 @@
 
 /* Set up the shadow-specific parts of a domain struct at start of day.
  * Called from paging_domain_init(). */
-int shadow_domain_init(struct domain *d, unsigned int domcr_flags);
+int shadow_domain_init(struct domain *d);
 
 /* Setup the shadow-specific parts of a vcpu struct. It is called by
  * paging_vcpu_init() in paging.c */
Signature · 7 lines
2.20.1.2.gb21ebb671


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v8 1/6
#2 lore raw

Re: [Xen-devel] [PATCH v8 1/6] x86/domain: remove the 'oos_off' flag

Quoted 4 lines · Paul Durrant

The flag is not needed since the domain 'options' can now be tested directly.

In principle

but

Julien, Stefano,

I'd like to to ask for an explicit opinion of at least one of you regarding the behavior on Arm. During v7 review I did suggest that the flag being set should get rejected there.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

Re: [Xen-devel] [PATCH v8 1/6] x86/domain: remove the 'oos_off' flag

Hi Jan,

Quoted 7 lines · Jan Beulich
Quoted 4 lines · Paul Durrant

The flag is not needed since the domain 'options' can now be tested directly.

In principle

Quoted 8 lines

but

Julien, Stefano,

I'd like to to ask for an explicit opinion of at least one of you regarding the behavior on Arm. During v7 review I did suggest that the flag being set should get rejected there.

The current code is actually rejecting any combination of flags but CDF_hvm_guest | CDF_hap. So the flag is effectively rejected for Arm.

However, it occurred to me that patch #2 will break domain creation on Arm as setting CDF_iommu will be prevented.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#5 lore raw

PATCH v8 2/6 domain: introduce XEN_DOMCTL_CDF_iommu flag

In reply to Paul Durrant

This patch introduces a common domain creation flag to determine whether the domain is permitted to make use of the IOMMU. Currently the flag is always set (for both dom0 and domU) if the IOMMU is globally enabled (i.e. iommu_enabled == 1). sanitise_domain_config() is modified to reject the flag if !iommu_enabled.

A new helper function, is_iommu_enabled(), is added to test the flag and iommu_domain_init() will return immediately if !is_iommu_enabled(). This is slightly different to the previous behaviour based on !iommu_enabled where the call to arch_iommu_domain_init() was made regardless, however it appears that this call was only necessary to initialize the dt_devices list for ARM such that iommu_release_dt_devices() can be called unconditionally by domain_relinquish_resources(). Adding a simple check of is_iommu_enabled() into iommu_release_dt_devices() keeps this unconditional call working.

No functional change should be observed with this patch applied.

Subsequent patches will allow the toolstack to control whether use of the IOMMU is enabled for a domain.

NOTE: The introduction of the is_iommu_enabled() helper function might
seem excessive but its use is expected to increase with subsequent patches. Also, having iommu_domain_init() bail before calling arch_iommu_domain_init() is not strictly necessary, but I think the consequent addition of the call to is_iommu_enabled() in iommu_release_dt_devices() makes the code clearer.

---
Cc: Christian Lindig <[email protected]>
Cc: David Scott <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>

Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html

v7:
- Add a check to verify that the toolstack has not set XEN_DOMCTL_CDF_iommu
- Add missing ocaml binding changes

v6:
- Remove the toolstack parts as there's no nice method of testing whether the IOMMU is enabled in an architecture-neutral way

v5:
- Move is_iommu_enabled() check into iommu_domain_init()
- Reject XEN_DOMCTL_CDF_iommu in sanitise_domain_config() if !iommu_enabled
- Use evaluate_nospec() in defintion of is_iommu_enabled()

---
 tools/ocaml/libs/xc/xenctrl.ml        |  8 +++++++-
 tools/ocaml/libs/xc/xenctrl.mli       |  8 +++++++-
 xen/arch/arm/setup.c                  |  3 +++
 xen/arch/x86/setup.c                  |  3 +++
 xen/common/domain.c                   |  9 ++++++++-
 xen/common/domctl.c                   | 13 +++++++++++++
 xen/drivers/passthrough/device_tree.c |  3 +++
 xen/drivers/passthrough/iommu.c       |  6 +++---
 xen/include/public/domctl.h           |  4 ++++
 xen/include/xen/sched.h               |  5 +++++
 10 files changed, 56 insertions(+), 6 deletions(-)
tools/ocaml/libs/xc/xenctrl.ml +7 -1
index 35958b94d5..bdf3f2e395 100644
@@ -56,7 +56,13 @@ type arch_domainconfig =
 	| ARM of xen_arm_arch_domainconfig
 	| X86 of xen_x86_arch_domainconfig
 
-type domain_create_flag = CDF_HVM | CDF_HAP
+type domain_create_flag =
+	| CDF_HVM
+	| CDF_HAP
+	| CDF_S3_INTEGRITY
+	| CDF_OOS_OFF
+	| CDF_XS_DOMAIN
+	| CDF_IOMMU
 
 type domctl_create_config =
 {
tools/ocaml/libs/xc/xenctrl.mli +7 -1
index 6c4268d453..fc40885671 100644
@@ -49,7 +49,13 @@ type arch_domainconfig =
   | ARM of xen_arm_arch_domainconfig
   | X86 of xen_x86_arch_domainconfig
 
-type domain_create_flag = CDF_HVM | CDF_HAP
+type domain_create_flag =
+  | CDF_HVM
+  | CDF_HAP
+  | CDF_S3_INTEGRITY
+  | CDF_OOS_OFF
+  | CDF_XS_DOMAIN
+  | CDF_IOMMU
 
 type domctl_create_config = {
   ssidref: int32;
xen/arch/arm/setup.c +3 -0
index fa6c110b11..dc2640f8b9 100644
@@ -961,6 +961,9 @@ void __init start_xen(unsigned long boot_phys_offset,
     dom0_cfg.arch.tee_type = tee_get_type();
     dom0_cfg.max_vcpus = dom0_max_vcpus();
 
+    if ( iommu_enabled )
+        dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
+
     dom0 = domain_create(0, &dom0_cfg, true);
     if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) )
         panic("Error creating domain 0\n");
xen/arch/x86/setup.c +3 -0
index d0b35b0ce2..fa226a2bab 100644
@@ -1733,6 +1733,9 @@ void __init noreturn __start_xen(unsigned long mbi_p)
     }
     dom0_cfg.max_vcpus = dom0_max_vcpus();
 
+    if ( iommu_enabled )
+        dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu;
+
     /* Create initial domain 0. */
     dom0 = domain_create(get_initial_domain_id(), &dom0_cfg, !pv_shim);
     if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) )
xen/common/domain.c +8 -1
index e9d2c613e0..7dfb257c50 100644
@@ -301,7 +301,8 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
                            XEN_DOMCTL_CDF_hap |
                            XEN_DOMCTL_CDF_s3_integrity |
                            XEN_DOMCTL_CDF_oos_off |
-                           XEN_DOMCTL_CDF_xs_domain) )
+                           XEN_DOMCTL_CDF_xs_domain |
+                           XEN_DOMCTL_CDF_iommu) )
     {
         dprintk(XENLOG_INFO, "Unknown CDF flags %#x\n", config->flags);
         return -EINVAL;
@@ -320,6 +321,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( (config->flags & XEN_DOMCTL_CDF_iommu) && !iommu_enabled )
+    {
+        dprintk(XENLOG_INFO, "IOMMU is not enabled\n");
+        return -EINVAL;
+    }
+
     return arch_sanitise_domain_config(config);
 }
 
xen/common/domctl.c +13 -0
index 6e6e9b9866..5dcfe3c8f6 100644
@@ -515,6 +515,19 @@ long do_domctl(XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
             rover = dom;
         }
 
+        /*
+         * For now, make sure the createdomain IOMMU flag is set if the
+         * IOMMU is enabled. When the flag comes under toolstack control
+         * this can go away.
+         */
+        if ( op->u.createdomain.flags & XEN_DOMCTL_CDF_iommu )
+        {
+            ASSERT_UNREACHABLE();
+            return -EINVAL;
+        }
+        if ( iommu_enabled )
+            op->u.createdomain.flags |= XEN_DOMCTL_CDF_iommu;
+
         d = domain_create(dom, &op->u.createdomain, false);
         if ( IS_ERR(d) )
         {
xen/drivers/passthrough/device_tree.c +3 -0
index b6eaae7283..d32b172664 100644
@@ -119,6 +119,9 @@ int iommu_release_dt_devices(struct domain *d)
     struct dt_device_node *dev, *_dev;
     int rc;
 
+    if ( !is_iommu_enabled(d) )
+        return 0;
+
     list_for_each_entry_safe(dev, _dev, &hd->dt_devices, domain_list)
     {
         rc = iommu_deassign_dt_device(d, dev);
xen/drivers/passthrough/iommu.c +3 -3
index 37eb0f7d01..e61d3d1368 100644
@@ -151,13 +151,13 @@ int iommu_domain_init(struct domain *d)
     struct domain_iommu *hd = dom_iommu(d);
     int ret = 0;
 
+    if ( !is_iommu_enabled(d) )
+        return 0;
+
     ret = arch_iommu_domain_init(d);
     if ( ret )
         return ret;
 
-    if ( !iommu_enabled )
-        return 0;
-
     hd->platform_ops = iommu_get_ops();
     return hd->platform_ops->init(d);
 }
xen/include/public/domctl.h +4 -0
index 72d5133cba..5f55a2f6e1 100644
@@ -64,6 +64,10 @@ struct xen_domctl_createdomain {
  /* Is this a xenstore domain? */
 #define _XEN_DOMCTL_CDF_xs_domain     4
 #define XEN_DOMCTL_CDF_xs_domain      (1U<<_XEN_DOMCTL_CDF_xs_domain)
+ /* Should this domain be permitted to use the IOMMU? */
+#define _XEN_DOMCTL_CDF_iommu         5
+#define XEN_DOMCTL_CDF_iommu          (1U<<_XEN_DOMCTL_CDF_iommu)
+
     uint32_t flags;
 
     /*
xen/include/xen/sched.h +5 -0
index d2bbe03bd9..3c0db42b82 100644
@@ -983,6 +983,11 @@ static inline bool is_xenstore_domain(const struct domain *d)
     return d->options & XEN_DOMCTL_CDF_xs_domain;
 }
 
+static inline bool is_iommu_enabled(const struct domain *d)
+{
+    return evaluate_nospec(d->options & XEN_DOMCTL_CDF_iommu);
+}
+
 extern bool sched_smt_power_savings;
 
 extern enum cpufreq_controller {
Signature · 7 lines
2.20.1.2.gb21ebb671


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#7 lore raw

Re: [Xen-devel] [PATCH v8 2/6] domain: introduce XEN_DOMCTL_CDF_iommu flag

In reply to Paul Durrant

Hi Paul,

Apologies for the late answer. A couple of comments below.

Quoted 5 lines · Paul Durrant

This patch introduces a common domain creation flag to determine whether the domain is permitted to make use of the IOMMU. Currently the flag is always set (for both dom0 and domU) if the IOMMU is globally enabled (i.e. iommu_enabled == 1). sanitise_domain_config() is modified to reject the flag if !iommu_enabled.

This is not entirely correct for Arm, the flag will not be set for domU created by Xen directly (i.e dom0less). Device passthrough is not yet supported for dom0less so this is not much a concern.

However, there are a series to add support for platform device passthrough (see [1]). So there are two possibilities:
1) If your series goes first, then we should at least document it in
the commit message that IOMMU will be disabled for domain (other than dom0) created by Xen.
2) If your series goes second, then you (or Stefano) would have to
ensure this does not break [1].

I haven't yet had the chance to review the latest version of Stefano and this patch looks in good shape. So 1) seems more suitable. Stefano would have to ensure the flags is set when doing device assignment.

Quoted 71 lines

A new helper function, is_iommu_enabled(), is added to test the flag and iommu_domain_init() will return immediately if !is_iommu_enabled(). This is slightly different to the previous behaviour based on !iommu_enabled where the call to arch_iommu_domain_init() was made regardless, however it appears that this call was only necessary to initialize the dt_devices list for ARM such that iommu_release_dt_devices() can be called unconditionally by domain_relinquish_resources(). Adding a simple check of is_iommu_enabled() into iommu_release_dt_devices() keeps this unconditional call working.

No functional change should be observed with this patch applied.

Subsequent patches will allow the toolstack to control whether use of the IOMMU is enabled for a domain.

NOTE: The introduction of the is_iommu_enabled() helper function might
seem excessive but its use is expected to increase with subsequent patches. Also, having iommu_domain_init() bail before calling arch_iommu_domain_init() is not strictly necessary, but I think the consequent addition of the call to is_iommu_enabled() in iommu_release_dt_devices() makes the code clearer.

---
Cc: Christian Lindig <[email protected]>
Cc: David Scott <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>

Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html

v7:
- Add a check to verify that the toolstack has not set XEN_DOMCTL_CDF_iommu
- Add missing ocaml binding changes

v6:
- Remove the toolstack parts as there's no nice method of testing whether the IOMMU is enabled in an architecture-neutral way

v5:
- Move is_iommu_enabled() check into iommu_domain_init()
- Reject XEN_DOMCTL_CDF_iommu in sanitise_domain_config() if !iommu_enabled
- Use evaluate_nospec() in defintion of is_iommu_enabled()

---
  tools/ocaml/libs/xc/xenctrl.ml        |  8 +++++++-
  tools/ocaml/libs/xc/xenctrl.mli       |  8 +++++++-
  xen/arch/arm/setup.c                  |  3 +++
  xen/arch/x86/setup.c                  |  3 +++
  xen/common/domain.c                   |  9 ++++++++-
  xen/common/domctl.c                   | 13 +++++++++++++
  xen/drivers/passthrough/device_tree.c |  3 +++
  xen/drivers/passthrough/iommu.c       |  6 +++---
  xen/include/public/domctl.h           |  4 ++++
  xen/include/xen/sched.h               |  5 +++++
  10 files changed, 56 insertions(+), 6 deletions(-)
tools/ocaml/libs/xc/xenctrl.ml +0 -0
index 35958b94d5..bdf3f2e395 100644
@@ -56,7 +56,13 @@ type arch_domainconfig =
  	| ARM of xen_arm_arch_domainconfig

tools/ocaml/libs/xc/xenctrl.ml type arch_domainconfig = · line 65

Quoted 10 lines
tools/ocaml/libs/xc/xenctrl.ml +7 -1
  	| X86 of xen_x86_arch_domainconfig
 
-type domain_create_flag = CDF_HVM | CDF_HAP
+type domain_create_flag =
+	| CDF_HVM
+	| CDF_HAP
+	| CDF_S3_INTEGRITY
+	| CDF_OOS_OFF
+	| CDF_XS_DOMAIN
+	| CDF_IOMMU

This patch is only adding the last flag, but you are adding more here. I understand that you are just re-syncing the type with Xen. IHMO, this should belong in a separate patch as we may want to backport it.

If backporting is not necessary, then the change should at least be mentioned in the commit message as this seems a bit off-topic.

Cheers,

[1] https://lists.xen.org/archives/html/xen-devel/2019-08/msg01974.html

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#8 lore raw

Re: [Xen-devel] [PATCH v8 2/6] domain: introduce XEN_DOMCTL_CDF_iommu flag

Quoted 1 line

-----Original Message-----

[snip]

Quoted 21 lines
Quoted 6 lines
tools/ocaml/libs/xc/xenctrl.ml +0 -0
index 35958b94d5..bdf3f2e395 100644
@@ -56,7 +56,13 @@ type arch_domainconfig =
  	| ARM of xen_arm_arch_domainconfig

tools/ocaml/libs/xc/xenctrl.ml type arch_domainconfig = · line 65

Quoted 10 lines
tools/ocaml/libs/xc/xenctrl.ml +7 -1
  	| X86 of xen_x86_arch_domainconfig
 
-type domain_create_flag = CDF_HVM | CDF_HAP
+type domain_create_flag =
+	| CDF_HVM
+	| CDF_HAP
+	| CDF_S3_INTEGRITY
+	| CDF_OOS_OFF
+	| CDF_XS_DOMAIN
+	| CDF_IOMMU

This patch is only adding the last flag, but you are adding more here. I understand that you are just re-syncing the type with Xen. IHMO, this should belong in a separate patch as we may want to backport it.

Quoted 3 lines

If backporting is not necessary, then the change should at least be mentioned in the commit message as this seems a bit off-topic.

The apparently extraneous flag additions are because of the way that the code at

http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=tools/ocaml/libs/xc/xenctrl_stubs.c;hb=HEAD#l149

works. TL;DR the index of the item in the list needs to match the bit position in the flags field.

Since you queried it I guess it'd better have a comment in the code.

Paul

Quoted 6 lines

Cheers,

[1] https://lists.xen.org/archives/html/xen-devel/2019-08/msg01974.html

--
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#9 lore raw

Re: [Xen-devel] [PATCH v8 2/6] domain: introduce XEN_DOMCTL_CDF_iommu flag

In reply to Paul Durrant

Hi,

Quoted 15 lines · Paul Durrant
xen/common/domain.c +2 -1
index e9d2c613e0..7dfb257c50 100644
@@ -301,7 +301,8 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
                             XEN_DOMCTL_CDF_hap |
                             XEN_DOMCTL_CDF_s3_integrity |
                             XEN_DOMCTL_CDF_oos_off |
-                           XEN_DOMCTL_CDF_xs_domain) )
+                           XEN_DOMCTL_CDF_xs_domain |
+                           XEN_DOMCTL_CDF_iommu) )
      {
          dprintk(XENLOG_INFO, "Unknown CDF flags %#x\n", config->flags);
          return -EINVAL;

xen/common/domain.c static int sanitise_domain_config(struct xen_domctl_createdomain *config) · line 329

Quoted 9 lines
xen/common/domain.c +6 -0
@@ -320,6 +321,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
          return -EINVAL;
      }
 
+    if ( (config->flags & XEN_DOMCTL_CDF_iommu) && !iommu_enabled )
+    {
+        dprintk(XENLOG_INFO, "IOMMU is not enabled\n");
+        return -EINVAL;
+    }
+

Looking at this patch again, the implementation of arch_sanitise_domain_config() for Arm will only accepts config->flags to be equal to CDF_hvm_guest | CDF_hap.

So after this patch, it will not be possible to create any domain when CDF_iommu is set.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#10 lore raw

Re: [Xen-devel] [PATCH v8 2/6] domain: introduce XEN_DOMCTL_CDF_iommu flag

Quoted 43 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 05 September 2019 21:06
To: Paul Durrant <[email protected]>; [email protected]
Cc: Roger Pau Monne <[email protected]>; Jan Beulich <[email protected]>; Christian Lindig <[email protected]>; David Scott <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Volodymyr Babchuk <[email protected]>
Subject: Re: [PATCH v8 2/6] domain: introduce XEN_DOMCTL_CDF_iommu flag

Hi,

Quoted 15 lines · Paul Durrant
xen/common/domain.c +2 -1
index e9d2c613e0..7dfb257c50 100644
@@ -301,7 +301,8 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
                             XEN_DOMCTL_CDF_hap |
                             XEN_DOMCTL_CDF_s3_integrity |
                             XEN_DOMCTL_CDF_oos_off |
-                           XEN_DOMCTL_CDF_xs_domain) )
+                           XEN_DOMCTL_CDF_xs_domain |
+                           XEN_DOMCTL_CDF_iommu) )
      {
          dprintk(XENLOG_INFO, "Unknown CDF flags %#x\n", config->flags);
          return -EINVAL;

xen/common/domain.c static int sanitise_domain_config(struct xen_domctl_createdomain *config) · line 329

Quoted 9 lines
xen/common/domain.c +6 -0
@@ -320,6 +321,12 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
          return -EINVAL;
      }
 
+    if ( (config->flags & XEN_DOMCTL_CDF_iommu) && !iommu_enabled )
+    {
+        dprintk(XENLOG_INFO, "IOMMU is not enabled\n");
+        return -EINVAL;
+    }
+

Looking at this patch again, the implementation of arch_sanitise_domain_config() for Arm will only accepts config->flags to be equal to CDF_hvm_guest | CDF_hap.

Quoted 2 lines

So after this patch, it will not be possible to create any domain when CDF_iommu is set.

You're right, I'm not sure how I missed that. I think I had changed it in development then managed to lose the hunk. Clearly ARM needs to accept the flag too.

Paul

Quoted 5 lines

Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#11 lore raw

PATCH v8 3/6 use is_iommu_enabled() where appropriate...

In reply to Paul Durrant

...rather than testing the global iommu_enabled flag and ops pointer.

Now that there is a per-domain flag indicating whether the domain is permitted to use the IOMMU (which determines whether the ops pointer will be set), many tests of the global iommu_enabled flag and ops pointer can be translated into tests of the per-domain flag. Some of the other tests of purely the global iommu_enabled flag can also be translated into tests of the per-domain flag.

NOTE: The comment in iommu_share_p2m_table() is also fixed; need_iommu()
disappeared some time ago. Also, whilst the style of the 'if' in flask_iommu_resource_use_perm() is fixed, I have not translated any instances of u32 into uint32_t to keep consistency. IMO such a translation would be better done globally for the source module in a separate patch.
The change to the definition of iommu_call() is to keep the PV shim build happy. Without this change it will fail to compile with errors of the form:

iommu.c:361:32: error: unused variable ‘hd’ [-Werror=unused-variable]

     const struct domain_iommu *hd = dom_iommu(d);
                                     ^~

---
Cc: Stefano Stabellini <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Jun Nakajima <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Suravee Suthikulpanit <[email protected]>

Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html

v7:
- Fix iommu_call() rather than messing with the initializtion of 'hd'
- Constify domain pointer passed to flask_iommu_resource_use_perm()

v5:
- Fix logic in ARM p2m_init()
- Make iommu_do_domctl() return -EOPNOTSUPP rather than -ENOSYS if the IOMMU is not enabled
- Fix test in pci_enable_acs()
- Fix test in flask_iommu_resource_use_perm()

---
 xen/arch/arm/p2m.c                        |  2 +-
 xen/arch/x86/dom0_build.c                 |  2 +-
 xen/arch/x86/domctl.c                     |  4 +--
 xen/arch/x86/hvm/hvm.c                    |  6 ++---
 xen/arch/x86/hvm/vioapic.c                |  2 +-
 xen/arch/x86/hvm/vmx/vmcs.c               |  2 +-
 xen/arch/x86/hvm/vmx/vmx.c                |  2 +-
 xen/arch/x86/mm/p2m-ept.c                 |  4 +--
 xen/drivers/passthrough/amd/iommu_guest.c |  2 +-
 xen/drivers/passthrough/device_tree.c     |  4 +--
 xen/drivers/passthrough/io.c              |  8 +++---
 xen/drivers/passthrough/iommu.c           | 31 ++++++++++-------------
 xen/drivers/passthrough/pci.c             | 16 ++++++------
 xen/drivers/passthrough/vtd/iommu.c       |  2 +-
 xen/drivers/passthrough/vtd/x86/hvm.c     |  2 +-
 xen/drivers/passthrough/x86/iommu.c       |  2 +-
 xen/include/asm-x86/iommu.h               | 11 ++++++--
 xen/xsm/flask/hooks.c                     | 18 ++++++-------
 18 files changed, 62 insertions(+), 58 deletions(-)
xen/arch/arm/p2m.c +1 -1
index e28ea1c85a..7f1442932a 100644
@@ -1531,7 +1531,7 @@ int p2m_init(struct domain *d)
      * shared with the CPU, Xen has to make sure that the PT changes have
      * reached the memory
      */
-    p2m->clean_pte = iommu_enabled &&
+    p2m->clean_pte = is_iommu_enabled(d) &&
         !iommu_has_feature(d, IOMMU_FEAT_COHERENT_WALK);
 
     rc = p2m_alloc_table(d);
xen/arch/x86/dom0_build.c +1 -1
index c69570920c..d381784edd 100644
@@ -356,7 +356,7 @@ unsigned long __init dom0_compute_nr_pages(
         avail -= d->max_vcpus - 1;
 
     /* Reserve memory for iommu_dom0_init() (rough estimate). */
-    if ( iommu_enabled )
+    if ( is_iommu_enabled(d) )
     {
         unsigned int s;
 
xen/arch/x86/domctl.c +2 -2
index 1e98fc8009..c4cb00bcf0 100644
@@ -703,7 +703,7 @@ long arch_do_domctl(
             break;
 
         ret = -ESRCH;
-        if ( iommu_enabled )
+        if ( is_iommu_enabled(d) )
         {
             pcidevs_lock();
             ret = pt_irq_create_bind(d, bind);
@@ -732,7 +732,7 @@ long arch_do_domctl(
         if ( ret )
             break;
 
-        if ( iommu_enabled )
+        if ( is_iommu_enabled(d) )
         {
             pcidevs_lock();
             ret = pt_irq_destroy_bind(d, bind);
xen/arch/x86/hvm/hvm.c +3 -3
index 029eea3b85..172c860acc 100644
@@ -465,7 +465,7 @@ void hvm_migrate_timers(struct vcpu *v)
 
 void hvm_migrate_pirq(struct hvm_pirq_dpci *pirq_dpci, const struct vcpu *v)
 {
-    ASSERT(iommu_enabled &&
+    ASSERT(is_iommu_enabled(v->domain) &&
            (is_hardware_domain(v->domain) || hvm_domain_irq(v->domain)->dpci));
 
     if ( (pirq_dpci->flags & HVM_IRQ_DPCI_MACH_MSI) &&
@@ -496,7 +496,7 @@ void hvm_migrate_pirqs(struct vcpu *v)
 {
     struct domain *d = v->domain;
 
-    if ( !iommu_enabled || !hvm_domain_irq(d)->dpci )
+    if ( !is_iommu_enabled(d) || !hvm_domain_irq(d)->dpci )
        return;
 
     spin_lock(&d->event_lock);
@@ -2264,7 +2264,7 @@ int hvm_set_cr0(unsigned long value, bool may_defer)
     }
 
     if ( ((value ^ old_value) & X86_CR0_CD) &&
-         iommu_enabled && hvm_funcs.handle_cd &&
+         is_iommu_enabled(d) && hvm_funcs.handle_cd &&
          (!rangeset_is_empty(d->iomem_caps) ||
           !rangeset_is_empty(d->arch.ioport_caps) ||
           has_arch_pdevs(d)) )
xen/arch/x86/hvm/vioapic.c +1 -1
index 9c25f72b4d..9aeef32a14 100644
@@ -536,7 +536,7 @@ void vioapic_update_EOI(struct domain *d, u8 vector)
 
             ent->fields.remote_irr = 0;
 
-            if ( iommu_enabled )
+            if ( is_iommu_enabled(d) )
             {
                 spin_unlock(&d->arch.hvm.irq_lock);
                 hvm_dpci_eoi(d, vioapic->base_gsi + pin, ent);
xen/arch/x86/hvm/vmx/vmcs.c +1 -1
index 16f14abe8f..ed27e8def7 100644
@@ -1087,7 +1087,7 @@ static int construct_vmcs(struct vcpu *v)
         vmx_clear_msr_intercept(v, MSR_IA32_SYSENTER_CS, VMX_MSR_RW);
         vmx_clear_msr_intercept(v, MSR_IA32_SYSENTER_ESP, VMX_MSR_RW);
         vmx_clear_msr_intercept(v, MSR_IA32_SYSENTER_EIP, VMX_MSR_RW);
-        if ( paging_mode_hap(d) && (!iommu_enabled || iommu_snoop) )
+        if ( paging_mode_hap(d) && (!is_iommu_enabled(d) || iommu_snoop) )
             vmx_clear_msr_intercept(v, MSR_IA32_CR_PAT, VMX_MSR_RW);
         if ( (vmexit_ctl & VM_EXIT_CLEAR_BNDCFGS) &&
              (vmentry_ctl & VM_ENTRY_LOAD_BNDCFGS) )
xen/arch/x86/hvm/vmx/vmx.c +1 -1
index 0060310d74..3b3d5b6250 100644
@@ -1210,7 +1210,7 @@ static void vmx_handle_cd(struct vcpu *v, unsigned long value)
         {
             v->arch.hvm.cache_mode = NORMAL_CACHE_MODE;
             vmx_set_guest_pat(v, *pat);
-            if ( !iommu_enabled || iommu_snoop )
+            if ( !is_iommu_enabled(v->domain) || iommu_snoop )
                 vmx_clear_msr_intercept(v, MSR_IA32_CR_PAT, VMX_MSR_RW);
             hvm_asid_flush_vcpu(v); /* no need to flush cache */
         }
xen/arch/x86/mm/p2m-ept.c +2 -2
index 952ebad82f..fa347e6026 100644
@@ -260,7 +260,7 @@ static bool_t ept_split_super_page(struct p2m_domain *p2m,
         *epte = *ept_entry;
         epte->sp = (level > 1);
         epte->mfn += i * trunk;
-        epte->snp = (iommu_enabled && iommu_snoop);
+        epte->snp = is_iommu_enabled(p2m->domain) && iommu_snoop;
         epte->suppress_ve = 1;
 
         ept_p2m_type_to_flags(p2m, epte, epte->sa_p2mt, epte->access);
@@ -766,7 +766,7 @@ ept_set_entry(struct p2m_domain *p2m, gfn_t gfn_, mfn_t mfn,
         new_entry.sp = !!i;
         new_entry.sa_p2mt = p2mt;
         new_entry.access = p2ma;
-        new_entry.snp = (iommu_enabled && iommu_snoop);
+        new_entry.snp = is_iommu_enabled(d) && iommu_snoop;
 
         /* the caller should take care of the previous page */
         new_entry.mfn = mfn_x(mfn);
xen/drivers/passthrough/amd/iommu_guest.c +1 -1
index 7f2dd662af..1f2bcfbe15 100644
@@ -821,7 +821,7 @@ int guest_iommu_init(struct domain* d)
     struct guest_iommu *iommu;
     struct domain_iommu *hd = dom_iommu(d);
 
-    if ( !is_hvm_domain(d) || !iommu_enabled || !iommuv2_enabled ||
+    if ( !is_hvm_domain(d) || !is_iommu_enabled(d) || !iommuv2_enabled ||
          !has_viommu(d) )
         return 0;
 
xen/drivers/passthrough/device_tree.c +2 -2
index d32b172664..12f2c4c3f2 100644
@@ -29,7 +29,7 @@ int iommu_assign_dt_device(struct domain *d, struct dt_device_node *dev)
     int rc = -EBUSY;
     struct domain_iommu *hd = dom_iommu(d);
 
-    if ( !iommu_enabled || !hd->platform_ops )
+    if ( !is_iommu_enabled(d) )
         return -EINVAL;
 
     if ( !dt_device_is_protected(dev) )
@@ -71,7 +71,7 @@ int iommu_deassign_dt_device(struct domain *d, struct dt_device_node *dev)
     const struct domain_iommu *hd = dom_iommu(d);
     int rc;
 
-    if ( !iommu_enabled || !hd->platform_ops )
+    if ( !is_iommu_enabled(d) )
         return -EINVAL;
 
     if ( !dt_device_is_protected(dev) )
xen/drivers/passthrough/io.c +4 -4
index 4290c7c710..b292e79382 100644
@@ -434,7 +434,7 @@ int pt_irq_create_bind(
             if ( vcpu )
                 pirq_dpci->gmsi.posted = true;
         }
-        if ( vcpu && iommu_enabled )
+        if ( vcpu && is_iommu_enabled(d) )
             hvm_migrate_pirq(pirq_dpci, vcpu);
 
         /* Use interrupt posting if it is supported. */
@@ -817,7 +817,7 @@ int hvm_do_IRQ_dpci(struct domain *d, struct pirq *pirq)
 
     ASSERT(is_hvm_domain(d));
 
-    if ( !iommu_enabled || (!is_hardware_domain(d) && !dpci) ||
+    if ( !is_iommu_enabled(d) || (!is_hardware_domain(d) && !dpci) ||
          !pirq_dpci || !(pirq_dpci->flags & HVM_IRQ_DPCI_MAPPED) )
         return 0;
 
@@ -869,7 +869,7 @@ static int _hvm_dpci_msi_eoi(struct domain *d,
 
 void hvm_dpci_msi_eoi(struct domain *d, int vector)
 {
-    if ( !iommu_enabled ||
+    if ( !is_iommu_enabled(d) ||
          (!hvm_domain_irq(d)->dpci && !is_hardware_domain(d)) )
        return;
 
@@ -1001,7 +1001,7 @@ void hvm_dpci_eoi(struct domain *d, unsigned int guest_gsi,
     const struct hvm_irq_dpci *hvm_irq_dpci;
     const struct hvm_girq_dpci_mapping *girq;
 
-    if ( !iommu_enabled )
+    if ( !is_iommu_enabled(d) )
         return;
 
     if ( is_hardware_domain(d) )
xen/drivers/passthrough/iommu.c +14 -17
index e61d3d1368..9dace64af9 100644
@@ -179,7 +179,7 @@ void __hwdom_init iommu_hwdom_init(struct domain *d)
 
     check_hwdom_reqs(d);
 
-    if ( !iommu_enabled )
+    if ( !is_iommu_enabled(d) )
         return;
 
     register_keyhandler('o', &iommu_dump_p2m_table, "dump iommu p2m table", 0);
@@ -284,7 +284,7 @@ int iommu_construct(struct domain *d)
 
 void iommu_domain_destroy(struct domain *d)
 {
-    if ( !iommu_enabled || !dom_iommu(d)->platform_ops )
+    if ( !is_iommu_enabled(d) )
         return;
 
     iommu_teardown(d);
@@ -300,7 +300,7 @@ int iommu_map(struct domain *d, dfn_t dfn, mfn_t mfn,
     unsigned long i;
     int rc = 0;
 
-    if ( !iommu_enabled || !hd->platform_ops )
+    if ( !is_iommu_enabled(d) )
         return 0;
 
     ASSERT(IS_ALIGNED(dfn_x(dfn), (1ul << page_order)));
@@ -360,7 +360,7 @@ int iommu_unmap(struct domain *d, dfn_t dfn, unsigned int page_order,
     unsigned long i;
     int rc = 0;
 
-    if ( !iommu_enabled || !hd->platform_ops )
+    if ( !is_iommu_enabled(d) )
         return 0;
 
     ASSERT(IS_ALIGNED(dfn_x(dfn), (1ul << page_order)));
@@ -413,7 +413,7 @@ int iommu_lookup_page(struct domain *d, dfn_t dfn, mfn_t *mfn,
 {
     const struct domain_iommu *hd = dom_iommu(d);
 
-    if ( !iommu_enabled || !hd->platform_ops || !hd->platform_ops->lookup_page )
+    if ( !is_iommu_enabled(d) || !hd->platform_ops->lookup_page )
         return -EOPNOTSUPP;
 
     return iommu_call(hd->platform_ops, lookup_page, d, dfn, mfn, flags);
@@ -442,8 +442,8 @@ int iommu_iotlb_flush(struct domain *d, dfn_t dfn, unsigned int page_count,
     const struct domain_iommu *hd = dom_iommu(d);
     int rc;
 
-    if ( !iommu_enabled || !hd->platform_ops ||
-         !hd->platform_ops->iotlb_flush || !page_count || !flush_flags )
+    if ( !is_iommu_enabled(d) || !hd->platform_ops->iotlb_flush ||
+         !page_count || !flush_flags )
         return 0;
 
     if ( dfn_eq(dfn, INVALID_DFN) )
@@ -470,8 +470,8 @@ int iommu_iotlb_flush_all(struct domain *d, unsigned int flush_flags)
     const struct domain_iommu *hd = dom_iommu(d);
     int rc;
 
-    if ( !iommu_enabled || !hd->platform_ops ||
-         !hd->platform_ops->iotlb_flush_all || !flush_flags )
+    if ( !is_iommu_enabled(d) || !hd->platform_ops->iotlb_flush_all ||
+         !flush_flags )
         return 0;
 
     /*
@@ -556,8 +556,8 @@ int iommu_do_domctl(
 {
     int ret = -ENODEV;
 
-    if ( !iommu_enabled )
-        return -ENOSYS;
+    if ( !is_iommu_enabled(d) )
+        return -EOPNOTSUPP;
 
 #ifdef CONFIG_HAS_PCI
     ret = iommu_do_pci_domctl(domctl, d, u_domctl);
@@ -576,9 +576,9 @@ void iommu_share_p2m_table(struct domain* d)
     ASSERT(hap_enabled(d));
     /*
      * iommu_use_hap_pt(d) cannot be used here because during domain
-     * construction need_iommu(d) will always return false here.
+     * construction has_iommu_pt(d) will always return false here.
      */
-    if ( iommu_enabled && iommu_hap_pt_share )
+    if ( is_iommu_enabled(d) && iommu_hap_pt_share )
         iommu_get_ops()->share_p2m(d);
 }
 
@@ -608,10 +608,7 @@ int iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
 
 bool_t iommu_has_feature(struct domain *d, enum iommu_feature feature)
 {
-    if ( !iommu_enabled )
-        return 0;
-
-    return test_bit(feature, dom_iommu(d)->features);
+    return is_iommu_enabled(d) && test_bit(feature, dom_iommu(d)->features);
 }
 
 static void iommu_dump_p2m_table(unsigned char key)
xen/drivers/passthrough/pci.c +8 -8
index af011d32dc..814106679f 100644
@@ -595,7 +595,7 @@ static void pci_enable_acs(struct pci_dev *pdev)
     u16 cap, ctrl, seg = pdev->seg;
     u8 bus = pdev->bus;
 
-    if ( !iommu_enabled )
+    if ( !is_iommu_enabled(pdev->domain) )
         return;
 
     pos = pci_find_ext_capability(seg, bus, pdev->devfn, PCI_EXT_CAP_ID_ACS);
@@ -864,7 +864,7 @@ static int pci_clean_dpci_irqs(struct domain *d)
 {
     struct hvm_irq_dpci *hvm_irq_dpci = NULL;
 
-    if ( !iommu_enabled )
+    if ( !is_iommu_enabled(d) )
         return 0;
 
     if ( !is_hvm_domain(d) )
@@ -897,7 +897,7 @@ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus,
     struct pci_dev *pdev;
     int ret = 0;
 
-    if ( !iommu_enabled || !hd->platform_ops )
+    if ( !is_iommu_enabled(d) )
         return -EINVAL;
 
     ASSERT(pcidevs_locked());
@@ -1383,7 +1383,7 @@ static int iommu_add_device(struct pci_dev *pdev)
     ASSERT(pcidevs_locked());
 
     hd = dom_iommu(pdev->domain);
-    if ( !iommu_enabled || !hd->platform_ops )
+    if ( !is_iommu_enabled(pdev->domain) )
         return 0;
 
     rc = hd->platform_ops->add_device(pdev->devfn, pci_to_dev(pdev));
@@ -1412,7 +1412,7 @@ static int iommu_enable_device(struct pci_dev *pdev)
     ASSERT(pcidevs_locked());
 
     hd = dom_iommu(pdev->domain);
-    if ( !iommu_enabled || !hd->platform_ops ||
+    if ( !is_iommu_enabled(pdev->domain) ||
          !hd->platform_ops->enable_device )
         return 0;
 
@@ -1428,7 +1428,7 @@ static int iommu_remove_device(struct pci_dev *pdev)
         return -EINVAL;
 
     hd = dom_iommu(pdev->domain);
-    if ( !iommu_enabled || !hd->platform_ops )
+    if ( !is_iommu_enabled(pdev->domain) )
         return 0;
 
     for ( devfn = pdev->devfn ; pdev->phantom_stride; )
@@ -1471,7 +1471,7 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag)
     struct pci_dev *pdev;
     int rc = 0;
 
-    if ( !iommu_enabled || !hd->platform_ops )
+    if ( !is_iommu_enabled(d) )
         return 0;
 
     /* Prevent device assign if mem paging or mem sharing have been 
@@ -1537,7 +1537,7 @@ static int iommu_get_device_group(
     int i = 0;
     const struct iommu_ops *ops = hd->platform_ops;
 
-    if ( !iommu_enabled || !ops || !ops->get_device_group_id )
+    if ( !is_iommu_enabled(d) || !ops->get_device_group_id )
         return 0;
 
     group_id = ops->get_device_group_id(seg, bus, devfn);
xen/drivers/passthrough/vtd/iommu.c +1 -1
index defa74fae3..e56d7befb4 100644
@@ -1757,7 +1757,7 @@ static void iommu_domain_teardown(struct domain *d)
         xfree(mrmrr);
     }
 
-    ASSERT(iommu_enabled);
+    ASSERT(is_iommu_enabled(d));
 
     /*
      * We can't use iommu_use_hap_pt here because either IOMMU state
xen/drivers/passthrough/vtd/x86/hvm.c +1 -1
index 6675dca027..f77b35815c 100644
@@ -51,7 +51,7 @@ void hvm_dpci_isairq_eoi(struct domain *d, unsigned int isairq)
     struct hvm_irq_dpci *dpci = NULL;
 
     ASSERT(isairq < NR_ISAIRQS);
-    if ( !iommu_enabled )
+    if ( !is_iommu_enabled(d) )
         return;
 
     spin_lock(&d->event_lock);
xen/drivers/passthrough/x86/iommu.c +1 -1
index 92c1d01edf..8319fe0a69 100644
@@ -180,7 +180,7 @@ int arch_iommu_populate_page_table(struct domain *d)
 
 void __hwdom_init arch_iommu_check_autotranslated_hwdom(struct domain *d)
 {
-    if ( !iommu_enabled )
+    if ( !is_iommu_enabled(d) )
         panic("Presently, iommu must be enabled for PVH hardware domain\n");
 }
 
xen/include/asm-x86/iommu.h +9 -2
index facf835ada..31fda4b0cf 100644
@@ -61,8 +61,15 @@ extern struct iommu_ops iommu_ops;
 
 #ifdef NDEBUG
 # include <asm/alternative.h>
-# define iommu_call(ops, fn, args...)  alternative_call(iommu_ops.fn, ## args)
-# define iommu_vcall(ops, fn, args...) alternative_vcall(iommu_ops.fn, ## args)
+# define iommu_call(ops, fn, args...) ({      \
+    (void)(ops);                              \
+    alternative_call(iommu_ops.fn, ## args);  \
+})
+
+# define iommu_vcall(ops, fn, args...) ({     \
+    (void)(ops);                              \
+    alternative_vcall(iommu_ops.fn, ## args); \
+})
 #endif
 
 static inline const struct iommu_ops *iommu_get_ops(void)
xen/xsm/flask/hooks.c +9 -9
index 6800f2d9a0..a449869550 100644
@@ -883,7 +883,7 @@ static int flask_map_domain_msi (struct domain *d, int irq, const void *data,
 #endif
 }
 
-static u32 flask_iommu_resource_use_perm(void)
+static u32 flask_iommu_resource_use_perm(const struct domain *d)
 {
     /* Obtain the permission level required for allowing a domain
      * to use an assigned device.
@@ -896,7 +896,7 @@ static u32 flask_iommu_resource_use_perm(void)
      */
     u32 perm = RESOURCE__USE_NOIOMMU;
 
-    if (iommu_enabled)
+    if ( is_iommu_enabled(d) )
         perm = ( iommu_intremap ? RESOURCE__USE_IOMMU :
                                   RESOURCE__USE_IOMMU_NOINTREMAP );
     return perm;
@@ -907,7 +907,7 @@ static int flask_map_domain_irq (struct domain *d, int irq, const void *data)
     u32 sid, dsid;
     int rc = -EPERM;
     struct avc_audit_data ad;
-    u32 dperm = flask_iommu_resource_use_perm();
+    u32 dperm = flask_iommu_resource_use_perm(d);
 
     if ( irq >= nr_static_irqs && data ) {
         rc = flask_map_domain_msi(d, irq, data, &sid, &ad);
@@ -973,7 +973,7 @@ static int flask_bind_pt_irq (struct domain *d, struct xen_domctl_bind_pt_irq *b
     int rc = -EPERM;
     int irq;
     struct avc_audit_data ad;
-    u32 dperm = flask_iommu_resource_use_perm();
+    u32 dperm = flask_iommu_resource_use_perm(d);
 
     rc = current_has_perm(d, SECCLASS_RESOURCE, RESOURCE__ADD);
     if ( rc )
@@ -1046,7 +1046,7 @@ static int flask_iomem_permission(struct domain *d, uint64_t start, uint64_t end
 
     data.ssid = domain_sid(current->domain);
     data.dsid = domain_sid(d);
-    data.use_perm = flask_iommu_resource_use_perm();
+    data.use_perm = flask_iommu_resource_use_perm(d);
 
     return security_iterate_iomem_sids(start, end, _iomem_has_perm, &data);
 }
@@ -1071,7 +1071,7 @@ static int flask_pci_config_permission(struct domain *d, uint32_t machine_bdf, u
     if ( access && (end >= 0x10 && start < 0x28) )
         perm = RESOURCE__SETUP;
     else
-        perm = flask_iommu_resource_use_perm();
+        perm = flask_iommu_resource_use_perm(d);
 
     AVC_AUDIT_DATA_INIT(&ad, DEV);
     ad.device = (unsigned long) machine_bdf;
@@ -1296,7 +1296,7 @@ static int flask_assign_device(struct domain *d, uint32_t machine_bdf)
     u32 dsid, rsid;
     int rc = -EPERM;
     struct avc_audit_data ad;
-    u32 dperm = flask_iommu_resource_use_perm();
+    u32 dperm = flask_iommu_resource_use_perm(d);
 
     if ( !d )
         return flask_test_assign_device(machine_bdf);
@@ -1355,7 +1355,7 @@ static int flask_assign_dtdevice(struct domain *d, const char *dtpath)
     u32 dsid, rsid;
     int rc = -EPERM;
     struct avc_audit_data ad;
-    u32 dperm = flask_iommu_resource_use_perm();
+    u32 dperm = flask_iommu_resource_use_perm(d);
 
     if ( !d )
         return flask_test_assign_dtdevice(dtpath);
@@ -1540,7 +1540,7 @@ static int flask_ioport_permission(struct domain *d, uint32_t start, uint32_t en
 
     data.ssid = domain_sid(current->domain);
     data.dsid = domain_sid(d);
-    data.use_perm = flask_iommu_resource_use_perm();
+    data.use_perm = flask_iommu_resource_use_perm(d);
 
     return security_iterate_ioport_sids(start, end, _ioport_has_perm, &data);
 }
Signature · 7 lines
2.20.1.2.gb21ebb671


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Acked-by on v8 3/6
#12 lore raw

Re: [Xen-devel] [PATCH v8 3/6] use is_iommu_enabled() where appropriate...

Hi Paul,

Quoted 20 lines · Paul Durrant

...rather than testing the global iommu_enabled flag and ops pointer.

Now that there is a per-domain flag indicating whether the domain is permitted to use the IOMMU (which determines whether the ops pointer will be set), many tests of the global iommu_enabled flag and ops pointer can be translated into tests of the per-domain flag. Some of the other tests of purely the global iommu_enabled flag can also be translated into tests of the per-domain flag.

NOTE: The comment in iommu_share_p2m_table() is also fixed; need_iommu()
disappeared some time ago. Also, whilst the style of the 'if' in flask_iommu_resource_use_perm() is fixed, I have not translated any instances of u32 into uint32_t to keep consistency. IMO such a translation would be better done globally for the source module in a separate patch.
The change to the definition of iommu_call() is to keep the PV shim build happy. Without this change it will fail to compile with errors of the form:

iommu.c:361:32: error: unused variable ‘hd’ [-Werror=unused-variable]

Quoted 8 lines
      const struct domain_iommu *hd = dom_iommu(d);
                                      ^~

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#13 lore raw

PATCH v8 4/6 remove late (on-demand) construction of IOMMU page tables

In reply to Paul Durrant

Now that there is a per-domain IOMMU-enable flag, which should be set if any device is going to be passed through, stop deferring page table construction until the assignment is done. Also don't tear down the tables again when the last device is de-assigned; defer that task until domain destruction.

This allows the has_iommu_pt() helper and iommu_status enumeration to be removed. Calls to has_iommu_pt() are simply replaced by calls to is_iommu_enabled(). Remaining open-coded tests of iommu_hap_pt_share can also be replaced by calls to iommu_use_hap_pt().
The arch_iommu_populate_page_table() and iommu_construct() functions become redundant, as does the 'strict mode' dom0 page_list mapping code in iommu_hwdom_init(), and iommu_teardown() can be made static is its only remaining caller, iommu_domain_destroy(), is within the same source module.

All in all, about 220 lines of code are removed from the hypervisor.

NOTE: This patch will cause a small amount of extra resource to be used
to accommodate IOMMU page tables that may never be used, since the per-domain IOMMU-enable flag is currently set to the value of the global iommu_enable flag. A subsequent patch will add an option to the toolstack to allow it to be turned off if there is no intention to assign passthrough hardware to the domain.
To account for the extra resource, 'iommu_memkb' has been added to domain_build_info. This patch sets it unconditionally to a value calculated based on the domain's maximum memory but, when the toolstack option mentioned above is added, it can be set to zero if the per-domain IOMMU-enable flag is turned off.

---
Cc: Stefano Stabellini <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Tamas K Lengyel <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Petre Pircalabu <[email protected]>

Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html

v7:
- Add toolstack memory reservation for IOMMU page tables... Re-use of shadow calculation didn't seem appropriate so a new helper function is added

v5:
- Minor style fixes

---
 tools/libxl/libxl_mem.c               |   6 +-
 tools/libxl/libxl_types.idl           |   1 +
 tools/libxl/libxl_utils.c             |  15 +++
 tools/libxl/libxl_utils.h             |   1 +
 tools/xl/xl_parse.c                   |   7 +-
 xen/arch/arm/p2m.c                    |   2 +-
 xen/arch/x86/dom0_build.c             |   2 +-
 xen/arch/x86/hvm/mtrr.c               |   5 +-
 xen/arch/x86/mm/mem_sharing.c         |   2 +-
 xen/arch/x86/mm/p2m.c                 |   4 +-
 xen/arch/x86/mm/paging.c              |   2 +-
 xen/arch/x86/x86_64/mm.c              |   2 +-
 xen/common/memory.c                   |   4 +-
 xen/common/vm_event.c                 |   2 +-
 xen/drivers/passthrough/device_tree.c |  11 ---
 xen/drivers/passthrough/iommu.c       | 134 ++++++--------------------
 xen/drivers/passthrough/pci.c         |  12 ---
 xen/drivers/passthrough/vtd/iommu.c   |  10 +-
 xen/drivers/passthrough/x86/iommu.c   |  97 -------------------
 xen/include/asm-arm/iommu.h           |   2 +-
 xen/include/asm-x86/iommu.h           |   2 +-
 xen/include/xen/iommu.h               |  16 ---
 xen/include/xen/sched.h               |   2 -
 23 files changed, 70 insertions(+), 271 deletions(-)
tools/libxl/libxl_mem.c +4 -2
index 448a2af8fd..fd6f33312e 100644
@@ -461,15 +461,17 @@ int libxl_domain_need_memory(libxl_ctx *ctx,
     if (rc) goto out;
 
     *need_memkb = b_info->target_memkb;
+    *need_memkb += b_info->shadow_memkb + b_info->iommu_memkb;
+
     switch (b_info->type) {
     case LIBXL_DOMAIN_TYPE_PVH:
     case LIBXL_DOMAIN_TYPE_HVM:
-        *need_memkb += b_info->shadow_memkb + LIBXL_HVM_EXTRA_MEMORY;
+        *need_memkb += LIBXL_HVM_EXTRA_MEMORY;
         if (libxl_defbool_val(b_info->device_model_stubdomain))
             *need_memkb += 32 * 1024;
         break;
     case LIBXL_DOMAIN_TYPE_PV:
-        *need_memkb += b_info->shadow_memkb + LIBXL_PV_EXTRA_MEMORY;
+        *need_memkb += LIBXL_PV_EXTRA_MEMORY;
         break;
     default:
         rc = ERROR_INVAL;
tools/libxl/libxl_types.idl +1 -0
index b61399ce36..d94b7453cb 100644
@@ -486,6 +486,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
     ("target_memkb",    MemKB),
     ("video_memkb",     MemKB),
     ("shadow_memkb",    MemKB),
+    ("iommu_memkb",     MemKB),
     ("rtc_timeoffset",  uint32),
     ("exec_ssidref",    uint32),
     ("exec_ssid_label", string),
tools/libxl/libxl_utils.c +15 -0
index f360f5e228..405733b7e1 100644
@@ -48,6 +48,21 @@ unsigned long libxl_get_required_shadow_memory(unsigned long maxmem_kb, unsigned
     return 4 * (256 * smp_cpus + 2 * (maxmem_kb / 1024));
 }
 
+unsigned long libxl_get_required_iommu_memory(unsigned long maxmem_kb)
+{
+    unsigned long iommu_pages = 0, mem_pages = maxmem_kb / 4;
+    unsigned int level;
+
+    /* Assume a 4 level page table with 512 entries per level */
+    for (level = 0; level < 4; level++)
+    {
+        mem_pages = DIV_ROUNDUP(mem_pages, 512);
+        iommu_pages += mem_pages;
+    }
+
+    return iommu_pages * 4;
+}
+
 char *libxl_domid_to_name(libxl_ctx *ctx, uint32_t domid)
 {
     unsigned int len;
tools/libxl/libxl_utils.h +1 -0
index 44409afdc4..630ccbe28a 100644
@@ -24,6 +24,7 @@ const
 char *libxl_basename(const char *name); /* returns string from strdup */
 
 unsigned long libxl_get_required_shadow_memory(unsigned long maxmem_kb, unsigned int smp_cpus);
+unsigned long libxl_get_required_iommu_memory(unsigned long maxmem_kb);
 int libxl_name_to_domid(libxl_ctx *ctx, const char *name, uint32_t *domid);
 int libxl_domain_qualifier_to_domid(libxl_ctx *ctx, const char *name, uint32_t *domid);
 char *libxl_domid_to_name(libxl_ctx *ctx, uint32_t domid);
tools/xl/xl_parse.c +5 -2
index e105bda2bb..c193fe9ba4 100644
@@ -1448,14 +1448,17 @@ void parse_config_data(const char *config_source,
         exit(1);
     }
 
-    /* libxl_get_required_shadow_memory() must be called after final values
+    /* libxl_get_required_shadow_memory() and
+     * libxl_get_required_iommu_memory() must be called after final values
      * (default or specified) for vcpus and memory are set, because the
-     * calculation depends on those values. */
+     * calculations depend on those values. */
     b_info->shadow_memkb = !xlu_cfg_get_long(config, "shadow_memory", &l, 0)
         ? l * 1024
         : libxl_get_required_shadow_memory(b_info->max_memkb,
                                            b_info->max_vcpus);
 
+    b_info->iommu_memkb = libxl_get_required_iommu_memory(b_info->max_memkb);
+
     xlu_cfg_get_defbool(config, "nomigrate", &b_info->disable_migrate, 0);
 
     if (!xlu_cfg_get_long(config, "tsc_mode", &l, 1)) {
xen/arch/arm/p2m.c +1 -1
index 7f1442932a..692565757e 100644
@@ -1056,7 +1056,7 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
          !mfn_eq(lpae_get_mfn(*entry), lpae_get_mfn(orig_pte)) )
         p2m_free_entry(p2m, orig_pte, level);
 
-    if ( has_iommu_pt(p2m->domain) &&
+    if ( is_iommu_enabled(p2m->domain) &&
          (lpae_is_valid(orig_pte) || lpae_is_valid(*entry)) )
     {
         unsigned int flush_flags = 0;
xen/arch/x86/dom0_build.c +1 -1
index d381784edd..7cfab2dc25 100644
@@ -365,7 +365,7 @@ unsigned long __init dom0_compute_nr_pages(
     }
 
     need_paging = is_hvm_domain(d) &&
-        (!iommu_hap_pt_share || !paging_mode_hap(d));
+        (!iommu_use_hap_pt(d) || !paging_mode_hap(d));
     for ( ; ; need_paging = false )
     {
         nr_pages = get_memsize(&dom0_size, avail);
xen/arch/x86/hvm/mtrr.c +3 -2
index 7ccd85bcea..5ad15eafe0 100644
@@ -783,7 +783,8 @@ HVM_REGISTER_SAVE_RESTORE(MTRR, hvm_save_mtrr_msr, hvm_load_mtrr_msr, 1,
 
 void memory_type_changed(struct domain *d)
 {
-    if ( (has_iommu_pt(d) || cache_flush_permitted(d)) && d->vcpu && d->vcpu[0] )
+    if ( (is_iommu_enabled(d) || cache_flush_permitted(d)) &&
+         d->vcpu && d->vcpu[0] )
     {
         p2m_memory_type_changed(d);
         flush_all(FLUSH_CACHE);
@@ -831,7 +832,7 @@ int epte_get_entry_emt(struct domain *d, unsigned long gfn, mfn_t mfn,
         return MTRR_TYPE_UNCACHABLE;
     }
 
-    if ( !has_iommu_pt(d) && !cache_flush_permitted(d) )
+    if ( !is_iommu_enabled(d) && !cache_flush_permitted(d) )
     {
         *ipat = 1;
         return MTRR_TYPE_WRBACK;
xen/arch/x86/mm/mem_sharing.c +1 -1
index a5fe89e339..efb8821768 100644
@@ -1664,7 +1664,7 @@ int mem_sharing_domctl(struct domain *d, struct xen_domctl_mem_sharing_op *mec)
         case XEN_DOMCTL_MEM_SHARING_CONTROL:
         {
             rc = 0;
-            if ( unlikely(has_iommu_pt(d) && mec->u.enable) )
+            if ( unlikely(is_iommu_enabled(d) && mec->u.enable) )
                 rc = -EXDEV;
             else
                 d->arch.hvm.mem_sharing_enabled = mec->u.enable;
xen/arch/x86/mm/p2m.c +2 -2
index 8a5229ee21..e5e4349dea 100644
@@ -1341,7 +1341,7 @@ int set_identity_p2m_entry(struct domain *d, unsigned long gfn_l,
 
     if ( !paging_mode_translate(p2m->domain) )
     {
-        if ( !has_iommu_pt(d) )
+        if ( !is_iommu_enabled(d) )
             return 0;
         return iommu_legacy_map(d, _dfn(gfn_l), _mfn(gfn_l), PAGE_ORDER_4K,
                                 IOMMUF_readable | IOMMUF_writable);
@@ -1432,7 +1432,7 @@ int clear_identity_p2m_entry(struct domain *d, unsigned long gfn_l)
 
     if ( !paging_mode_translate(d) )
     {
-        if ( !has_iommu_pt(d) )
+        if ( !is_iommu_enabled(d) )
             return 0;
         return iommu_legacy_unmap(d, _dfn(gfn_l), PAGE_ORDER_4K);
     }
xen/arch/x86/mm/paging.c +1 -1
index 69aa228e46..d9a52c4db4 100644
@@ -213,7 +213,7 @@ int paging_log_dirty_enable(struct domain *d, bool_t log_global)
 {
     int ret;
 
-    if ( has_iommu_pt(d) && log_global )
+    if ( is_iommu_enabled(d) && log_global )
     {
         /*
          * Refuse to turn on global log-dirty mode
xen/arch/x86/x86_64/mm.c +1 -1
index 795a467462..fa55f3474e 100644
@@ -1434,7 +1434,7 @@ int memory_add(unsigned long spfn, unsigned long epfn, unsigned int pxm)
      * shared or being kept in sync then newly added memory needs to be
      * mapped here.
      */
-    if ( has_iommu_pt(hardware_domain) &&
+    if ( is_iommu_enabled(hardware_domain) &&
          !iommu_use_hap_pt(hardware_domain) &&
          !need_iommu_pt_sync(hardware_domain) )
     {
xen/common/memory.c +2 -2
index d5aff83f2d..7364fd2c33 100644
@@ -823,7 +823,7 @@ int xenmem_add_to_physmap(struct domain *d, struct xen_add_to_physmap *xatp,
     xatp->gpfn += start;
     xatp->size -= start;
 
-    if ( has_iommu_pt(d) )
+    if ( is_iommu_enabled(d) )
        this_cpu(iommu_dont_flush_iotlb) = 1;
 
     while ( xatp->size > done )
@@ -844,7 +844,7 @@ int xenmem_add_to_physmap(struct domain *d, struct xen_add_to_physmap *xatp,
         }
     }
 
-    if ( has_iommu_pt(d) )
+    if ( is_iommu_enabled(d) )
     {
         int ret;
 
xen/common/vm_event.c +1 -1
index 2a1c87e44b..3b18195ebf 100644
@@ -630,7 +630,7 @@ int vm_event_domctl(struct domain *d, struct xen_domctl_vm_event_op *vec)
 
             /* No paging if iommu is used */
             rc = -EMLINK;
-            if ( unlikely(has_iommu_pt(d)) )
+            if ( unlikely(is_iommu_enabled(d)) )
                 break;
 
             rc = -EXDEV;
xen/drivers/passthrough/device_tree.c +0 -11
index 12f2c4c3f2..ea9fd54e3b 100644
@@ -40,17 +40,6 @@ int iommu_assign_dt_device(struct domain *d, struct dt_device_node *dev)
     if ( !list_empty(&dev->domain_list) )
         goto fail;
 
-    /*
-     * The hwdom is forced to use IOMMU for protecting assigned
-     * device. Therefore the IOMMU data is already set up.
-     */
-    ASSERT(!is_hardware_domain(d) ||
-           hd->status == IOMMU_STATUS_initialized);
-
-    rc = iommu_construct(d);
-    if ( rc )
-        goto fail;
-
     /* The flag field doesn't matter to DT device. */
     rc = hd->platform_ops->assign_device(d, 0, dt_to_dev(dev), 0);
 
xen/drivers/passthrough/iommu.c +28 -106

182 lines not on this page · raw message

xen/drivers/passthrough/pci.c +0 -12
index 814106679f..2315d490dc 100644
@@ -933,9 +933,6 @@ static int deassign_device(struct domain *d, uint16_t seg, uint8_t bus,
 
     pdev->fault.count = 0;
 
-    if ( !has_arch_pdevs(d) && has_iommu_pt(d) )
-        iommu_teardown(d);
-
     return ret;
 }
 
@@ -1484,13 +1481,6 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag)
     if ( !pcidevs_trylock() )
         return -ERESTART;
 
-    rc = iommu_construct(d);
-    if ( rc )
-    {
-        pcidevs_unlock();
-        return rc;
-    }
-
     pdev = pci_get_pdev_by_domain(hardware_domain, seg, bus, devfn);
     if ( !pdev )
     {
@@ -1519,8 +1509,6 @@ static int assign_device(struct domain *d, u16 seg, u8 bus, u8 devfn, u32 flag)
     }
 
  done:
-    if ( !has_arch_pdevs(d) && has_iommu_pt(d) )
-        iommu_teardown(d);
     pcidevs_unlock();
 
     return rc;
xen/drivers/passthrough/vtd/iommu.c +1 -9
index e56d7befb4..ae4608fe14 100644
@@ -1759,15 +1759,7 @@ static void iommu_domain_teardown(struct domain *d)
 
     ASSERT(is_iommu_enabled(d));
 
-    /*
-     * We can't use iommu_use_hap_pt here because either IOMMU state
-     * is already changed to IOMMU_STATUS_disabled at this point or
-     * has always been IOMMU_STATUS_disabled.
-     *
-     * We also need to test if HAP is enabled because PV guests can
-     * enter this path too.
-     */
-    if ( hap_enabled(d) && iommu_hap_pt_share )
+    if ( iommu_use_hap_pt(d) )
         return;
 
     spin_lock(&hd->arch.mapping_lock);
xen/drivers/passthrough/x86/iommu.c +0 -97

103 lines not on this page · raw message

xen/include/asm-arm/iommu.h +1 -1
index 904c9aec11..1577e83d2b 100644
@@ -21,7 +21,7 @@ struct arch_iommu
 };
 
 /* Always share P2M Table between the CPU and the IOMMU */
-#define iommu_use_hap_pt(d) (has_iommu_pt(d))
+#define iommu_use_hap_pt(d) is_iommu_enabled(d)
 
 const struct iommu_ops *iommu_get_ops(void);
 void iommu_set_ops(const struct iommu_ops *ops);
xen/include/asm-x86/iommu.h +1 -1
index 31fda4b0cf..5071afd6a5 100644
@@ -88,7 +88,7 @@ extern const struct iommu_init_ops *iommu_init_ops;
 
 /* Are we using the domain P2M table as its IOMMU pagetable? */
 #define iommu_use_hap_pt(d) \
-    (hap_enabled(d) && has_iommu_pt(d) && iommu_hap_pt_share)
+    (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share)
 
 void iommu_update_ire_from_apic(unsigned int apic, unsigned int reg, unsigned int value);
 unsigned int iommu_read_apic_from_ire(unsigned int apic, unsigned int reg);
xen/include/xen/iommu.h +0 -16
index 314f28f323..4b6871936c 100644
@@ -73,15 +73,9 @@ void iommu_domain_destroy(struct domain *d);
 
 void arch_iommu_domain_destroy(struct domain *d);
 int arch_iommu_domain_init(struct domain *d);
-int arch_iommu_populate_page_table(struct domain *d);
 void arch_iommu_check_autotranslated_hwdom(struct domain *d);
 void arch_iommu_hwdom_init(struct domain *d);
 
-int iommu_construct(struct domain *d);
-
-/* Function used internally, use iommu_domain_destroy */
-void iommu_teardown(struct domain *d);
-
 /*
  * The following flags are passed to map operations and passed by lookup
  * operations.
@@ -248,13 +242,6 @@ struct iommu_ops {
 # define iommu_vcall iommu_call
 #endif
 
-enum iommu_status
-{
-    IOMMU_STATUS_disabled,
-    IOMMU_STATUS_initializing,
-    IOMMU_STATUS_initialized
-};
-
 struct domain_iommu {
     struct arch_iommu arch;
 
@@ -269,9 +256,6 @@ struct domain_iommu {
     /* Features supported by the IOMMU */
     DECLARE_BITMAP(features, IOMMU_FEAT_count);
 
-    /* Status of guest IOMMU mappings */
-    enum iommu_status status;
-
     /*
      * Does the guest reqire mappings to be synchonized, to maintain
      * the default dfn == pfn map. (See comment on dfn at the top of
xen/include/xen/sched.h +0 -2
index 3c0db42b82..3f8ad56655 100644
@@ -966,10 +966,8 @@ static inline bool is_hwdom_pinned_vcpu(const struct vcpu *v)
 }
 
 #ifdef CONFIG_HAS_PASSTHROUGH
-#define has_iommu_pt(d) (dom_iommu(d)->status != IOMMU_STATUS_disabled)
 #define need_iommu_pt_sync(d) (dom_iommu(d)->need_sync)
 #else
-#define has_iommu_pt(d) false
 #define need_iommu_pt_sync(d) false
 #endif
 
Signature · 7 lines
2.20.1.2.gb21ebb671


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#14 lore raw

Re: [Xen-devel] [PATCH v8 4/6] remove late (on-demand) construction of IOMMU page tables

Quoted 30 lines · Paul Durrant

Now that there is a per-domain IOMMU-enable flag, which should be set if any device is going to be passed through, stop deferring page table construction until the assignment is done. Also don't tear down the tables again when the last device is de-assigned; defer that task until domain destruction.

This allows the has_iommu_pt() helper and iommu_status enumeration to be removed. Calls to has_iommu_pt() are simply replaced by calls to is_iommu_enabled(). Remaining open-coded tests of iommu_hap_pt_share can also be replaced by calls to iommu_use_hap_pt().
The arch_iommu_populate_page_table() and iommu_construct() functions become redundant, as does the 'strict mode' dom0 page_list mapping code in iommu_hwdom_init(), and iommu_teardown() can be made static is its only remaining caller, iommu_domain_destroy(), is within the same source module.

All in all, about 220 lines of code are removed from the hypervisor.

NOTE: This patch will cause a small amount of extra resource to be used
to accommodate IOMMU page tables that may never be used, since the per-domain IOMMU-enable flag is currently set to the value of the global iommu_enable flag. A subsequent patch will add an option to the toolstack to allow it to be turned off if there is no intention to assign passthrough hardware to the domain.
To account for the extra resource, 'iommu_memkb' has been added to domain_build_info. This patch sets it unconditionally to a value calculated based on the domain's maximum memory but, when the toolstack option mentioned above is added, it can be set to zero if the per-domain IOMMU-enable flag is turned off.

Quoted 1 line

I've just stumbled across our earlier discussion on the still not sufficiently "x86/HVM: p2m_ram_ro is incompatible with device pass-through" of mine, and I wonder whether the implication from the change here is that people wanting p2m_ram_ro used on a guest would then need to force the IOMMU off for that guest (which aiui isn't possible until patch 6).

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#15 lore raw

Re: [Xen-devel] [PATCH v8 4/6] remove late (on-demand) construction of IOMMU page tables

Quoted 44 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 05 September 2019 15:30
To: Paul Durrant <[email protected]>
Cc: [email protected]; Julien Grall <[email protected]>; Alexandru Isaila <[email protected]>; PetrePircalabu <[email protected]>; Razvan Cojocaru <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monne <[email protected]>; Volodymyr Babchuk <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Stefano Stabellini <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Tamas K Lengyel <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH v8 4/6] remove late (on-demand) construction of IOMMU page tables

Quoted 30 lines · Paul Durrant

Now that there is a per-domain IOMMU-enable flag, which should be set if any device is going to be passed through, stop deferring page table construction until the assignment is done. Also don't tear down the tables again when the last device is de-assigned; defer that task until domain destruction.

This allows the has_iommu_pt() helper and iommu_status enumeration to be removed. Calls to has_iommu_pt() are simply replaced by calls to is_iommu_enabled(). Remaining open-coded tests of iommu_hap_pt_share can also be replaced by calls to iommu_use_hap_pt().
The arch_iommu_populate_page_table() and iommu_construct() functions become redundant, as does the 'strict mode' dom0 page_list mapping code in iommu_hwdom_init(), and iommu_teardown() can be made static is its only remaining caller, iommu_domain_destroy(), is within the same source module.

All in all, about 220 lines of code are removed from the hypervisor.

NOTE: This patch will cause a small amount of extra resource to be used
to accommodate IOMMU page tables that may never be used, since the per-domain IOMMU-enable flag is currently set to the value of the global iommu_enable flag. A subsequent patch will add an option to the toolstack to allow it to be turned off if there is no intention to assign passthrough hardware to the domain.
To account for the extra resource, 'iommu_memkb' has been added to domain_build_info. This patch sets it unconditionally to a value calculated based on the domain's maximum memory but, when the toolstack option mentioned above is added, it can be set to zero if the per-domain IOMMU-enable flag is turned off.

Quoted 8 lines
Quoted 1 line

I've just stumbled across our earlier discussion on the still not sufficiently "x86/HVM: p2m_ram_ro is incompatible with device pass-through" of mine, and I wonder whether the implication from the change here is that people wanting p2m_ram_ro used on a guest would then need to force the IOMMU off for that guest (which aiui isn't possible until patch 6).

You wouldn't be able to force IOMMU off on a per-domain basis until patch #6 but it could still be done globally.

Paul

Quoted 2 lines

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#16 lore raw

Re: [Xen-devel] [PATCH v8 4/6] remove late (on-demand) construction of IOMMU page tables

In reply to Paul Durrant

Hi,

tools/libxl/libxl_mem.c int libxl_domain_need_memory(libxl_ctx *ctx, · line 464

Quoted 9 lines · Paul Durrant
tools/libxl/libxl_mem.c +1 -0
index 448a2af8fd..fd6f33312e 100644
@@ -461,15 +461,17 @@ int libxl_domain_need_memory(libxl_ctx *ctx,
      if (rc) goto out;
 
      *need_memkb = b_info->target_memkb;
+    *need_memkb += b_info->shadow_memkb + b_info->iommu_memkb;

AFAICT, iommu_memkb will be non-0 even when the IOMMU share the page-table with the CPUs. If so, why is this required for that case?

Quoted 20 lines
tools/libxl/libxl_mem.c +3 -2
+
      switch (b_info->type) {
      case LIBXL_DOMAIN_TYPE_PVH:
      case LIBXL_DOMAIN_TYPE_HVM:
-        *need_memkb += b_info->shadow_memkb + LIBXL_HVM_EXTRA_MEMORY;
+        *need_memkb += LIBXL_HVM_EXTRA_MEMORY;
          if (libxl_defbool_val(b_info->device_model_stubdomain))
              *need_memkb += 32 * 1024;
          break;
      case LIBXL_DOMAIN_TYPE_PV:
-        *need_memkb += b_info->shadow_memkb + LIBXL_PV_EXTRA_MEMORY;
+        *need_memkb += LIBXL_PV_EXTRA_MEMORY;
          break;
      default:
          rc = ERROR_INVAL;
tools/libxl/libxl_types.idl +0 -0
index b61399ce36..d94b7453cb 100644

tools/libxl/libxl_types.idl libxl_domain_build_info = Struct("domain_build_info",[ · line 489

Quoted 4 lines
tools/libxl/libxl_types.idl +1 -0
@@ -486,6 +486,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
      ("target_memkb",    MemKB),
      ("video_memkb",     MemKB),
      ("shadow_memkb",    MemKB),
+    ("iommu_memkb",     MemKB),

I think you want a corresponding LIBXL_HAVE in libxl.h to tell external toolstack whether the field exist.

tools/libxl/libxl_types.idl libxl_domain_build_info = Struct("domain_build_info",[ · line 492

Quoted 3 lines
tools/libxl/libxl_types.idl +0 -0
      ("rtc_timeoffset",  uint32),
      ("exec_ssidref",    uint32),
      ("exec_ssid_label", string),

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#17 lore raw

Re: [Xen-devel] [PATCH v8 4/6] remove late (on-demand) construction of IOMMU page tables

Quoted 23 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 05 September 2019 21:21
To: Paul Durrant <[email protected]>; [email protected]
Cc: Alexandru Isaila <[email protected]>; Razvan Cojocaru <[email protected]>; Jan Beulich <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monne <[email protected]>; Tamas K Lengyel <[email protected]>; Petre Pircalabu <[email protected]>
Subject: Re: [PATCH v8 4/6] remove late (on-demand) construction of IOMMU page tables

Hi,

Quoted 7 lines · Paul Durrant
tools/libxl/libxl_mem.c +0 -0
index 448a2af8fd..fd6f33312e 100644
@@ -461,15 +461,17 @@ int libxl_domain_need_memory(libxl_ctx *ctx,
      if (rc) goto out;
 
Quoted 5 lines

tools/libxl/libxl_mem.c int libxl_domain_need_memory(libxl_ctx *ctx, · line 464

Quoted 2 lines
tools/libxl/libxl_mem.c +1 -0
      *need_memkb = b_info->target_memkb;
+    *need_memkb += b_info->shadow_memkb + b_info->iommu_memkb;

AFAICT, iommu_memkb will be non-0 even when the IOMMU share the page-table with the CPUs. If so, why is this required for that case?

The toostack can't know about shared EPT as there's no mechanism to tell it. Once patch #6 goes in though, the toolstack will be able to select shared and forego the overhead. However, I've just realized that of course this means that the domain may fail due to lack of resources on a host which doesn't support shared EPT so I think I'm going to have to add add extra info (following on from Roger's recent patch) so the toolstack can know whether shared EPT is available.

Paul

Quoted 29 lines
Quoted 21 lines
tools/libxl/libxl_mem.c +3 -2
+
      switch (b_info->type) {
      case LIBXL_DOMAIN_TYPE_PVH:
      case LIBXL_DOMAIN_TYPE_HVM:
-        *need_memkb += b_info->shadow_memkb + LIBXL_HVM_EXTRA_MEMORY;
+        *need_memkb += LIBXL_HVM_EXTRA_MEMORY;
          if (libxl_defbool_val(b_info->device_model_stubdomain))
              *need_memkb += 32 * 1024;
          break;
      case LIBXL_DOMAIN_TYPE_PV:
-        *need_memkb += b_info->shadow_memkb + LIBXL_PV_EXTRA_MEMORY;
+        *need_memkb += LIBXL_PV_EXTRA_MEMORY;
          break;
      default:
          rc = ERROR_INVAL;
tools/libxl/libxl_types.idl +0 -0
index b61399ce36..d94b7453cb 100644

tools/libxl/libxl_types.idl libxl_domain_build_info = Struct("domain_build_info",[ · line 489

Quoted 4 lines
tools/libxl/libxl_types.idl +1 -0
@@ -486,6 +486,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
      ("target_memkb",    MemKB),
      ("video_memkb",     MemKB),
      ("shadow_memkb",    MemKB),
+    ("iommu_memkb",     MemKB),

I think you want a corresponding LIBXL_HAVE in libxl.h to tell external toolstack whether the field exist.

Quoted 8 lines

tools/libxl/libxl_types.idl libxl_domain_build_info = Struct("domain_build_info",[ · line 492

Quoted 3 lines
tools/libxl/libxl_types.idl +0 -0
      ("rtc_timeoffset",  uint32),
      ("exec_ssidref",    uint32),
      ("exec_ssid_label", string),

Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#18 lore raw

PATCH v8 5/6 iommu: tidy up iommu_use_hap_pt() and need_iommu_pt_sync() macros

In reply to Paul Durrant

Thes macros really ought to live in the common xen/iommu.h header rather then being distributed amongst architecture specific iommu headers and xen/sched.h. This patch moves them there.

NOTE: Disabling 'sharept' in the command line iommu options should really
be hard error on ARM (as opposed to just being ignored), so define 'iommu_hap_pt_share' to be true for ARM then then gate parsing the command line option on '#ifndef iommu_hap_pt_share'.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

Previously part of https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html

v7:
- Re-work the ARM handling of 'sharept' as suggested by Jan
- Make sure that need_iommu_pt_sync() always evaluates its argument

---
 xen/drivers/passthrough/iommu.c |  8 +++++++-
 xen/include/asm-arm/iommu.h     |  3 ---
 xen/include/asm-x86/iommu.h     |  4 ----
 xen/include/xen/iommu.h         | 19 ++++++++++++++++++-
 xen/include/xen/sched.h         |  6 ------
 5 files changed, 25 insertions(+), 15 deletions(-)
xen/drivers/passthrough/iommu.c +7 -1
index 4f71db95ea..aaf3b9fac0 100644
@@ -49,7 +49,11 @@ int8_t __hwdom_initdata iommu_hwdom_reserved = -1;
  * default until we find a good solution to resolve it.
  */
 bool_t __read_mostly iommu_intpost;
-bool_t __read_mostly iommu_hap_pt_share = 1;
+
+#ifndef iommu_hap_pt_share
+bool __read_mostly iommu_hap_pt_share = true;
+#endif
+
 bool_t __read_mostly iommu_debug;
 bool_t __read_mostly amd_iommu_perdev_intremap = 1;
 
@@ -102,8 +106,10 @@ static int __init parse_iommu_param(const char *s)
             iommu_hwdom_passthrough = val;
         else if ( (val = parse_boolean("dom0-strict", s, ss)) >= 0 )
             iommu_hwdom_strict = val;
+#ifndef iommu_hap_pt_share
         else if ( (val = parse_boolean("sharept", s, ss)) >= 0 )
             iommu_hap_pt_share = val;
+#endif
         else
             rc = -EINVAL;
 
xen/include/asm-arm/iommu.h +0 -3
index 1577e83d2b..77a94b29eb 100644
@@ -20,9 +20,6 @@ struct arch_iommu
     void *priv;
 };
 
-/* Always share P2M Table between the CPU and the IOMMU */
-#define iommu_use_hap_pt(d) is_iommu_enabled(d)
-
 const struct iommu_ops *iommu_get_ops(void);
 void iommu_set_ops(const struct iommu_ops *ops);
 
xen/include/asm-x86/iommu.h +0 -4
index 5071afd6a5..85741f7c96 100644
@@ -86,10 +86,6 @@ struct iommu_init_ops {
 
 extern const struct iommu_init_ops *iommu_init_ops;
 
-/* Are we using the domain P2M table as its IOMMU pagetable? */
-#define iommu_use_hap_pt(d) \
-    (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share)
-
 void iommu_update_ire_from_apic(unsigned int apic, unsigned int reg, unsigned int value);
 unsigned int iommu_read_apic_from_ire(unsigned int apic, unsigned int reg);
 int iommu_setup_hpet_msi(struct msi_desc *);
xen/include/xen/iommu.h +18 -1
index 4b6871936c..87f9129b99 100644
@@ -55,7 +55,13 @@ static inline bool_t dfn_eq(dfn_t x, dfn_t y)
 extern bool_t iommu_enable, iommu_enabled;
 extern bool_t force_iommu, iommu_verbose, iommu_igfx;
 extern bool_t iommu_snoop, iommu_qinval, iommu_intremap, iommu_intpost;
-extern bool_t iommu_hap_pt_share;
+
+#ifdef CONFIG_ARM
+#define iommu_hap_pt_share true
+#else
+extern bool iommu_hap_pt_share;
+#endif
+
 extern bool_t iommu_debug;
 extern bool_t amd_iommu_perdev_intremap;
 
@@ -268,6 +274,17 @@ struct domain_iommu {
 #define iommu_set_feature(d, f)   set_bit(f, dom_iommu(d)->features)
 #define iommu_clear_feature(d, f) clear_bit(f, dom_iommu(d)->features)
 
+/* Are we using the domain P2M table as its IOMMU pagetable? */
+#define iommu_use_hap_pt(d) \
+    (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share)
+
+/* Does the IOMMU pagetable need to be kept synchronized with the P2M */
+#ifdef CONFIG_HAS_PASSTHROUGH
+#define need_iommu_pt_sync(d)     (dom_iommu(d)->need_sync)
+#else
+#define need_iommu_pt_sync(d)     ({ (void)(d); false; })
+#endif
+
 int __must_check iommu_suspend(void);
 void iommu_resume(void);
 void iommu_crash_shutdown(void);
xen/include/xen/sched.h +0 -6
index 3f8ad56655..0b5c106a37 100644
@@ -965,12 +965,6 @@ static inline bool is_hwdom_pinned_vcpu(const struct vcpu *v)
             cpumask_weight(v->cpu_hard_affinity) == 1);
 }
 
-#ifdef CONFIG_HAS_PASSTHROUGH
-#define need_iommu_pt_sync(d) (dom_iommu(d)->need_sync)
-#else
-#define need_iommu_pt_sync(d) false
-#endif
-
 static inline bool is_vcpu_online(const struct vcpu *v)
 {
     return !test_bit(_VPF_down, &v->pause_flags);
Signature · 7 lines
2.20.1.2.gb21ebb671


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#19 lore raw

Re: [Xen-devel] [PATCH v8 5/6] iommu: tidy up iommu_use_hap_pt() and need_iommu_pt_sync() macros

Hi Paul,

Quoted 98 lines · Paul Durrant

Thes macros really ought to live in the common xen/iommu.h header rather then being distributed amongst architecture specific iommu headers and xen/sched.h. This patch moves them there.

NOTE: Disabling 'sharept' in the command line iommu options should really
be hard error on ARM (as opposed to just being ignored), so define 'iommu_hap_pt_share' to be true for ARM then then gate parsing the command line option on '#ifndef iommu_hap_pt_share'.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

Previously part of https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html

v7:
- Re-work the ARM handling of 'sharept' as suggested by Jan
- Make sure that need_iommu_pt_sync() always evaluates its argument

---
  xen/drivers/passthrough/iommu.c |  8 +++++++-
  xen/include/asm-arm/iommu.h     |  3 ---
  xen/include/asm-x86/iommu.h     |  4 ----
  xen/include/xen/iommu.h         | 19 ++++++++++++++++++-
  xen/include/xen/sched.h         |  6 ------
  5 files changed, 25 insertions(+), 15 deletions(-)
xen/drivers/passthrough/iommu.c +7 -1
index 4f71db95ea..aaf3b9fac0 100644
@@ -49,7 +49,11 @@ int8_t __hwdom_initdata iommu_hwdom_reserved = -1;
   * default until we find a good solution to resolve it.
   */
  bool_t __read_mostly iommu_intpost;
-bool_t __read_mostly iommu_hap_pt_share = 1;
+
+#ifndef iommu_hap_pt_share
+bool __read_mostly iommu_hap_pt_share = true;
+#endif
+
  bool_t __read_mostly iommu_debug;
  bool_t __read_mostly amd_iommu_perdev_intremap = 1;
 
@@ -102,8 +106,10 @@ static int __init parse_iommu_param(const char *s)
              iommu_hwdom_passthrough = val;
          else if ( (val = parse_boolean("dom0-strict", s, ss)) >= 0 )
              iommu_hwdom_strict = val;
+#ifndef iommu_hap_pt_share
          else if ( (val = parse_boolean("sharept", s, ss)) >= 0 )
              iommu_hap_pt_share = val;
+#endif
          else
              rc = -EINVAL;
 
xen/include/asm-arm/iommu.h +0 -3
index 1577e83d2b..77a94b29eb 100644
@@ -20,9 +20,6 @@ struct arch_iommu
      void *priv;
  };
 
-/* Always share P2M Table between the CPU and the IOMMU */
-#define iommu_use_hap_pt(d) is_iommu_enabled(d)
-
  const struct iommu_ops *iommu_get_ops(void);
  void iommu_set_ops(const struct iommu_ops *ops);
 
xen/include/asm-x86/iommu.h +0 -4
index 5071afd6a5..85741f7c96 100644
@@ -86,10 +86,6 @@ struct iommu_init_ops {
 
  extern const struct iommu_init_ops *iommu_init_ops;
 
-/* Are we using the domain P2M table as its IOMMU pagetable? */
-#define iommu_use_hap_pt(d) \
-    (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share)
-
  void iommu_update_ire_from_apic(unsigned int apic, unsigned int reg, unsigned int value);
  unsigned int iommu_read_apic_from_ire(unsigned int apic, unsigned int reg);
  int iommu_setup_hpet_msi(struct msi_desc *);
xen/include/xen/iommu.h +0 -0
index 4b6871936c..87f9129b99 100644

xen/include/xen/iommu.h static inline bool_t dfn_eq(dfn_t x, dfn_t y) · line 63

Quoted 10 lines
xen/include/xen/iommu.h +6 -1
@@ -55,7 +55,13 @@ static inline bool_t dfn_eq(dfn_t x, dfn_t y)
  extern bool_t iommu_enable, iommu_enabled;
  extern bool_t force_iommu, iommu_verbose, iommu_igfx;
  extern bool_t iommu_snoop, iommu_qinval, iommu_intremap, iommu_intpost;
-extern bool_t iommu_hap_pt_share;
+
+#ifdef CONFIG_ARM
+#define iommu_hap_pt_share true
+#else
+extern bool iommu_hap_pt_share;
+#endif

I don't particularly like #ifdef CONFIG_<ARCH> in common header. How about other arch? I can see two solutions:

1) Move the define in asm/iommu.h. This would require to move the declaration a bit later and then protect as you did in iommu.c
2) Replace CONFIG_ARM with a new Kconfig selected by Arm only so far.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#20 lore raw

Re: [Xen-devel] [PATCH v8 5/6] iommu: tidy up iommu_use_hap_pt() and need_iommu_pt_sync() macros

Quoted 126 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 05 September 2019 20:38
To: Paul Durrant <[email protected]>; [email protected]
Cc: Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [PATCH v8 5/6] iommu: tidy up iommu_use_hap_pt() and need_iommu_pt_sync() macros

Hi Paul,

Quoted 98 lines · Paul Durrant

Thes macros really ought to live in the common xen/iommu.h header rather then being distributed amongst architecture specific iommu headers and xen/sched.h. This patch moves them there.

NOTE: Disabling 'sharept' in the command line iommu options should really
be hard error on ARM (as opposed to just being ignored), so define 'iommu_hap_pt_share' to be true for ARM then then gate parsing the command line option on '#ifndef iommu_hap_pt_share'.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

Previously part of https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html

v7:
- Re-work the ARM handling of 'sharept' as suggested by Jan
- Make sure that need_iommu_pt_sync() always evaluates its argument

---
  xen/drivers/passthrough/iommu.c |  8 +++++++-
  xen/include/asm-arm/iommu.h     |  3 ---
  xen/include/asm-x86/iommu.h     |  4 ----
  xen/include/xen/iommu.h         | 19 ++++++++++++++++++-
  xen/include/xen/sched.h         |  6 ------
  5 files changed, 25 insertions(+), 15 deletions(-)
xen/drivers/passthrough/iommu.c +7 -1
index 4f71db95ea..aaf3b9fac0 100644
@@ -49,7 +49,11 @@ int8_t __hwdom_initdata iommu_hwdom_reserved = -1;
   * default until we find a good solution to resolve it.
   */
  bool_t __read_mostly iommu_intpost;
-bool_t __read_mostly iommu_hap_pt_share = 1;
+
+#ifndef iommu_hap_pt_share
+bool __read_mostly iommu_hap_pt_share = true;
+#endif
+
  bool_t __read_mostly iommu_debug;
  bool_t __read_mostly amd_iommu_perdev_intremap = 1;
 
@@ -102,8 +106,10 @@ static int __init parse_iommu_param(const char *s)
              iommu_hwdom_passthrough = val;
          else if ( (val = parse_boolean("dom0-strict", s, ss)) >= 0 )
              iommu_hwdom_strict = val;
+#ifndef iommu_hap_pt_share
          else if ( (val = parse_boolean("sharept", s, ss)) >= 0 )
              iommu_hap_pt_share = val;
+#endif
          else
              rc = -EINVAL;
 
xen/include/asm-arm/iommu.h +0 -3
index 1577e83d2b..77a94b29eb 100644
@@ -20,9 +20,6 @@ struct arch_iommu
      void *priv;
  };
 
-/* Always share P2M Table between the CPU and the IOMMU */
-#define iommu_use_hap_pt(d) is_iommu_enabled(d)
-
  const struct iommu_ops *iommu_get_ops(void);
  void iommu_set_ops(const struct iommu_ops *ops);
 
xen/include/asm-x86/iommu.h +0 -4
index 5071afd6a5..85741f7c96 100644
@@ -86,10 +86,6 @@ struct iommu_init_ops {
 
  extern const struct iommu_init_ops *iommu_init_ops;
 
-/* Are we using the domain P2M table as its IOMMU pagetable? */
-#define iommu_use_hap_pt(d) \
-    (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share)
-
  void iommu_update_ire_from_apic(unsigned int apic, unsigned int reg, unsigned int value);
  unsigned int iommu_read_apic_from_ire(unsigned int apic, unsigned int reg);
  int iommu_setup_hpet_msi(struct msi_desc *);
xen/include/xen/iommu.h +0 -0
index 4b6871936c..87f9129b99 100644

xen/include/xen/iommu.h static inline bool_t dfn_eq(dfn_t x, dfn_t y) · line 63

Quoted 10 lines
xen/include/xen/iommu.h +6 -1
@@ -55,7 +55,13 @@ static inline bool_t dfn_eq(dfn_t x, dfn_t y)
  extern bool_t iommu_enable, iommu_enabled;
  extern bool_t force_iommu, iommu_verbose, iommu_igfx;
  extern bool_t iommu_snoop, iommu_qinval, iommu_intremap, iommu_intpost;
-extern bool_t iommu_hap_pt_share;
+
+#ifdef CONFIG_ARM
+#define iommu_hap_pt_share true
+#else
+extern bool iommu_hap_pt_share;
+#endif

I don't particularly like #ifdef CONFIG_<ARCH> in common header. How about other arch? I can see two solutions:

Quoted 4 lines

1) Move the define in asm/iommu.h. This would require to move the declaration a bit later and then protect as you did in iommu.c
2) Replace CONFIG_ARM with a new Kconfig selected by Arm only so far.

I had wondered about a Kconfig but I can't really think of a good name. How about CONFIG_FORCE_PT_SHARE?

Paul

Quoted 4 lines

Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#21 lore raw

Re: [Xen-devel] [PATCH v8 5/6] iommu: tidy up iommu_use_hap_pt() and need_iommu_pt_sync() macros

Hi Paul,

Quoted 126 lines · Paul Durrant
Quoted 126 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 05 September 2019 20:38
To: Paul Durrant <[email protected]>; [email protected]
Cc: Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [PATCH v8 5/6] iommu: tidy up iommu_use_hap_pt() and need_iommu_pt_sync() macros

Hi Paul,

Quoted 98 lines · Paul Durrant

Thes macros really ought to live in the common xen/iommu.h header rather then being distributed amongst architecture specific iommu headers and xen/sched.h. This patch moves them there.

NOTE: Disabling 'sharept' in the command line iommu options should really
be hard error on ARM (as opposed to just being ignored), so define 'iommu_hap_pt_share' to be true for ARM then then gate parsing the command line option on '#ifndef iommu_hap_pt_share'.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

Previously part of https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html

v7:
- Re-work the ARM handling of 'sharept' as suggested by Jan
- Make sure that need_iommu_pt_sync() always evaluates its argument

---
   xen/drivers/passthrough/iommu.c |  8 +++++++-
   xen/include/asm-arm/iommu.h     |  3 ---
   xen/include/asm-x86/iommu.h     |  4 ----
   xen/include/xen/iommu.h         | 19 ++++++++++++++++++-
   xen/include/xen/sched.h         |  6 ------
   5 files changed, 25 insertions(+), 15 deletions(-)
xen/drivers/passthrough/iommu.c +7 -1
index 4f71db95ea..aaf3b9fac0 100644
@@ -49,7 +49,11 @@ int8_t __hwdom_initdata iommu_hwdom_reserved = -1;
    * default until we find a good solution to resolve it.
    */
   bool_t __read_mostly iommu_intpost;
-bool_t __read_mostly iommu_hap_pt_share = 1;
+
+#ifndef iommu_hap_pt_share
+bool __read_mostly iommu_hap_pt_share = true;
+#endif
+
   bool_t __read_mostly iommu_debug;
   bool_t __read_mostly amd_iommu_perdev_intremap = 1;
 
@@ -102,8 +106,10 @@ static int __init parse_iommu_param(const char *s)
               iommu_hwdom_passthrough = val;
           else if ( (val = parse_boolean("dom0-strict", s, ss)) >= 0 )
               iommu_hwdom_strict = val;
+#ifndef iommu_hap_pt_share
           else if ( (val = parse_boolean("sharept", s, ss)) >= 0 )
               iommu_hap_pt_share = val;
+#endif
           else
               rc = -EINVAL;
 
xen/include/asm-arm/iommu.h +0 -3
index 1577e83d2b..77a94b29eb 100644
@@ -20,9 +20,6 @@ struct arch_iommu
       void *priv;
   };
 
-/* Always share P2M Table between the CPU and the IOMMU */
-#define iommu_use_hap_pt(d) is_iommu_enabled(d)
-
   const struct iommu_ops *iommu_get_ops(void);
   void iommu_set_ops(const struct iommu_ops *ops);
 
xen/include/asm-x86/iommu.h +0 -4
index 5071afd6a5..85741f7c96 100644
@@ -86,10 +86,6 @@ struct iommu_init_ops {
 
   extern const struct iommu_init_ops *iommu_init_ops;
 
-/* Are we using the domain P2M table as its IOMMU pagetable? */
-#define iommu_use_hap_pt(d) \
-    (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share)
-
   void iommu_update_ire_from_apic(unsigned int apic, unsigned int reg, unsigned int value);
   unsigned int iommu_read_apic_from_ire(unsigned int apic, unsigned int reg);
   int iommu_setup_hpet_msi(struct msi_desc *);
xen/include/xen/iommu.h +0 -0
index 4b6871936c..87f9129b99 100644

xen/include/xen/iommu.h static inline bool_t dfn_eq(dfn_t x, dfn_t y) · line 63

Quoted 10 lines
xen/include/xen/iommu.h +6 -1
@@ -55,7 +55,13 @@ static inline bool_t dfn_eq(dfn_t x, dfn_t y)
   extern bool_t iommu_enable, iommu_enabled;
   extern bool_t force_iommu, iommu_verbose, iommu_igfx;
   extern bool_t iommu_snoop, iommu_qinval, iommu_intremap, iommu_intpost;
-extern bool_t iommu_hap_pt_share;
+
+#ifdef CONFIG_ARM
+#define iommu_hap_pt_share true
+#else
+extern bool iommu_hap_pt_share;
+#endif

I don't particularly like #ifdef CONFIG_<ARCH> in common header. How about other arch? I can see two solutions:

Quoted 6 lines
Quoted 4 lines

1) Move the define in asm/iommu.h. This would require to move the declaration a bit later and then protect as you did in iommu.c
2) Replace CONFIG_ARM with a new Kconfig selected by Arm only so far.

I had wondered about a Kconfig but I can't really think of a good name. How about CONFIG_FORCE_PT_SHARE?

I would add "IOMMU" in the name just to make clear this is related to IOMMU. So maybe CONFIG_IOMMU_FORCE_PT_SHARE.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#22 lore raw

PATCH v8 6/6 introduce a 'passthrough' configuration option to xl.cfg...

In reply to Paul Durrant

...and hence the ability to disable IOMMU mappings, and control EPT sharing.

This patch introduces a new 'libxl_passthrough' enumeration into libxl_domain_create_info. The value will be set by xl either when it parses a new 'passthrough' option in xl.cfg, or implicitly if there is passthrough hardware specified for the domain.

If the value of the passthrough configuration option is 'disabled' then the XEN_DOMCTL_CDF_iommu flag will be clear in the xen_domctl_createdomain flags, thus allowing the toolstack to control whether the domain gets IOMMU mappings or not (where previously they were globally set).

If the value of the passthrough configuration option is 'sync_pt' then a new 'iommu_opts' field in xen_domctl_createdomain will be set with the value XEN_DOMCTL_IOMMU_no_sharept. This will override the global default set in iommu_hap_pt_share, thus allowing the toolstack to control whether EPT sharing is used for the domain.

NOTE: The 'iommu_memkb' overhead in libxl_domain_build_info will only be
set to zero if passthrough is 'disabled'. It is not safe to set the overhead to zero in the 'share_pt' case because the toolstack has no means of knowing whether the hardware actually supports IOMMU page table sharing.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Christian Lindig <[email protected]>
Cc: David Scott <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html

v7:
- Added missing breaks
- Added missing ocaml binding changes

v6:
- Remove the libxl_physinfo() call since it's usefulness is limited to x86

v5:
- Expand xen_domctl_createdomain flags field and hence bump interface version
- Fix spelling mistakes in context line

---
 docs/man/xl.cfg.5.pod.in            |  52 +++++++++++
 tools/libxl/libxl.h                 |   5 +
 tools/libxl/libxl_create.c          |   9 ++
 tools/libxl/libxl_types.idl         |   7 ++
 tools/ocaml/libs/xc/xenctrl.ml      |   4 +
 tools/ocaml/libs/xc/xenctrl.mli     |   4 +
 tools/ocaml/libs/xc/xenctrl_stubs.c |  15 ++-
 tools/xl/xl_parse.c                 | 140 ++++++++++++++++++----------
 xen/arch/arm/domain.c               |  10 +-
 xen/arch/x86/domain.c               |   2 +-
 xen/common/domain.c                 |   7 ++
 xen/common/domctl.c                 |  13 ---
 xen/drivers/passthrough/iommu.c     |  13 ++-
 xen/include/public/domctl.h         |   6 +-
 xen/include/xen/iommu.h             |  19 ++--
 15 files changed, 229 insertions(+), 77 deletions(-)
docs/man/xl.cfg.5.pod.in +52 -0
index c99d40307e..fd35685e9e 100644
@@ -605,6 +605,58 @@ option should only be used with a trusted device tree.
 Note that the partial device tree should avoid using the phandle 65000
 which is reserved by the toolstack.
 
+=item B<passthrough="STRING">
+
+Specify whether IOMMU mappings are enabled for the domain and hence whether
+it will be enabled for passthrough hardware. Valid values for this option
+are:
+
+=over 4
+
+=item B<disabled>
+
+IOMMU mappings are disabled for the domain and so hardware may not be
+passed through.
+
+This option is the default if no passthrough hardware is specified
+in the domain's configuration.
+
+=item B<sync_pt>
+
+This option means that IOMMU mappings will be synchronized with the
+domain's P2M table as follows:
+
+For a PV domain, all writable pages assigned to the domain are identity
+mapped by MFN in the IOMMU page table. Thus a device driver running in the
+domain may program passthrough hardware for DMA using MFN values
+(i.e. host/machine frame numbers) looked up in its P2M.
+
+For an HVM domain, all non-foreign RAM pages present in its P2M will be
+mapped by GFN in the IOMMU page table. Thus a device driver running in the
+domain may program passthrough hardware using GFN values (i.e. guest
+physical frame numbers) without any further translation.
+
+This option is the default if the domain is PV and passthrough hardware
+is specified in the configuration.
+
+This option is not available on Arm.
+
+=item B<share_pt>
+
+This option is unavailable for a PV domain. For an HVM domain, this option
+means that the IOMMU will be programmed to directly reference the domain's
+P2M table as its page table. From the point of view of a device driver
+running in the domain this is functionally equivalent to B<sync_pt> but
+places less load on the hypervisor and so should generally be selected in
+preference. However, the availability of this option is hardware specific
+and thus, if it is specified for a domain running on hardware that does
+not allow it (e.g. AMD), B<sync_pt> will be used instead.
+
+This option is the default if the domain is HVM and passthrough hardware
+is specified in the configuration.
+
+=back
+
 =back
 
 =head2 Devices
tools/libxl/libxl.h +5 -0
index 9bacfb97f0..5de7c07a41 100644
@@ -394,6 +394,11 @@
  */
 #define LIBXL_HAVE_EXTENDED_VKB 1
 
+/*
+ * libxl_domain_create_info has libxl_passthrough enumeration.
+ */
+#define LIBXL_HAVE_CREATEINFO_PASSTHROUGH 1
+
 /*
  * libxl ABI compatibility
  *
tools/libxl/libxl_create.c +9 -0
index 03ce166f4f..f288e13dc1 100644
@@ -564,6 +564,15 @@ int libxl__domain_make(libxl__gc *gc, libxl_domain_config *d_config,
                 libxl_defbool_val(info->oos) ? 0 : XEN_DOMCTL_CDF_oos_off;
         }
 
+        LOG(DETAIL, "passthrough: %s",
+            libxl_passthrough_to_string(info->passthrough));
+
+        if (info->passthrough != LIBXL_PASSTHROUGH_DISABLED)
+            create.flags |= XEN_DOMCTL_CDF_iommu;
+
+        if (info->passthrough == LIBXL_PASSTHROUGH_SYNC_PT)
+            create.iommu_opts |= XEN_DOMCTL_IOMMU_no_sharept;
+
         /* Ultimately, handle is an array of 16 uint8_t, same as uuid */
         libxl_uuid_copy(ctx, (libxl_uuid *)&create.handle, &info->uuid);
 
tools/libxl/libxl_types.idl +7 -0
index d94b7453cb..4ee4fc3dad 100644
@@ -263,6 +263,12 @@ libxl_vkb_backend = Enumeration("vkb_backend", [
     (2, "LINUX")
     ])
 
+libxl_passthrough = Enumeration("passthrough", [
+    (0, "disabled"),
+    (1, "sync_pt"),
+    (2, "share_pt"),
+    ])
+
 #
 # Complex libxl types
 #
@@ -408,6 +414,7 @@ libxl_domain_create_info = Struct("domain_create_info",[
     ("pool_name",    string),
     ("run_hotplug_scripts",libxl_defbool),
     ("driver_domain",libxl_defbool),
+    ("passthrough",  libxl_passthrough),
     ], dir=DIR_IN)
 
 libxl_domain_restore_params = Struct("domain_restore_params", [
tools/ocaml/libs/xc/xenctrl.ml +4 -0
index bdf3f2e395..4834a391fd 100644
@@ -64,11 +64,15 @@ type domain_create_flag =
 	| CDF_XS_DOMAIN
 	| CDF_IOMMU
 
+type domain_create_iommu_opts =
+	| IOMMU_NO_SHAREPT
+
 type domctl_create_config =
 {
 	ssidref: int32;
 	handle: string;
 	flags: domain_create_flag list;
+	iommu_opts: domain_create_iommu_opts list;
 	max_vcpus: int;
 	max_evtchn_port: int;
 	max_grant_frames: int;
tools/ocaml/libs/xc/xenctrl.mli +4 -0
index fc40885671..5d9548cc0d 100644
@@ -57,10 +57,14 @@ type domain_create_flag =
   | CDF_XS_DOMAIN
   | CDF_IOMMU
 
+type domain_create_iommu_opts =
+  | IOMMU_NO_SHAREPT
+
 type domctl_create_config = {
   ssidref: int32;
   handle: string;
   flags: domain_create_flag list;
+  iommu_opts: domain_create_iommu_opts list;
   max_vcpus: int;
   max_evtchn_port: int;
   max_grant_frames: int;
tools/ocaml/libs/xc/xenctrl_stubs.c +10 -5
index 2e1b29ce33..31b03e9494 100644
@@ -128,11 +128,12 @@ CAMLprim value stub_xc_domain_create(value xch, value config)
 #define VAL_SSIDREF             Field(config, 0)
 #define VAL_HANDLE              Field(config, 1)
 #define VAL_FLAGS               Field(config, 2)
-#define VAL_MAX_VCPUS           Field(config, 3)
-#define VAL_MAX_EVTCHN_PORT     Field(config, 4)
-#define VAL_MAX_GRANT_FRAMES    Field(config, 5)
-#define VAL_MAX_MAPTRACK_FRAMES Field(config, 6)
-#define VAL_ARCH                Field(config, 7)
+#define VAL_IOMMU_OPTS          Field(config, 3)
+#define VAL_MAX_VCPUS           Field(config, 4)
+#define VAL_MAX_EVTCHN_PORT     Field(config, 5)
+#define VAL_MAX_GRANT_FRAMES    Field(config, 6)
+#define VAL_MAX_MAPTRACK_FRAMES Field(config, 7)
+#define VAL_ARCH                Field(config, 8)
 
 	uint32_t domid = 0;
 	int result;
@@ -149,6 +150,9 @@ CAMLprim value stub_xc_domain_create(value xch, value config)
 	for ( l = VAL_FLAGS; l != Val_none; l = Field(l, 1) )
 		cfg.flags |= 1u << Int_val(Field(l, 0));
 
+	for ( l = VAL_IOMMU_OPTS; l != Val_none; l = Field(l, 1) )
+		cfg.iommu_opts |= 1u << Int_val(Field(l, 0));
+
 	arch_domconfig = Field(VAL_ARCH, 0);
 	switch ( Tag_val(VAL_ARCH) )
 	{
@@ -181,6 +185,7 @@ CAMLprim value stub_xc_domain_create(value xch, value config)
 #undef VAL_MAX_GRANT_FRAMES
 #undef VAL_MAX_EVTCHN_PORT
 #undef VAL_MAX_VCPUS
+#undef VAL_IOMMU_OPTS
 #undef VAL_FLAGS
 #undef VAL_HANDLE
 #undef VAL_SSIDREF
tools/xl/xl_parse.c +91 -49

158 lines not on this page · raw message

xen/arch/arm/domain.c +9 -1
index 941bbff4fe..b12de6ff3d 100644
@@ -614,6 +614,14 @@ int arch_sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    /* Always share P2M Table between the CPU and the IOMMU */
+    if ( config->iommu_opts & XEN_DOMCTL_IOMMU_no_sharept )
+    {
+        dprintk(XENLOG_INFO,
+                "Unsupported iommu option: XEN_DOMCTL_IOMMU_no_sharept\n");
+        return -EINVAL;
+    }
+
     /* Fill in the native GIC version, passed back to the toolstack. */
     if ( config->arch.gic_version == XEN_DOMCTL_CONFIG_GIC_NATIVE )
     {
@@ -674,7 +682,7 @@ int arch_domain_create(struct domain *d,
     ASSERT(config != NULL);
 
     /* p2m_init relies on some value initialized by the IOMMU subsystem */
-    if ( (rc = iommu_domain_init(d)) != 0 )
+    if ( (rc = iommu_domain_init(d, config->iommu_opts)) != 0 )
         goto fail;
 
     if ( (rc = p2m_init(d)) != 0 )
xen/arch/x86/domain.c +1 -1
index d5a19404a6..50d53ee878 100644
@@ -611,7 +611,7 @@ int arch_domain_create(struct domain *d,
     if ( (rc = init_domain_irq_mapping(d)) != 0 )
         goto fail;
 
-    if ( (rc = iommu_domain_init(d)) != 0 )
+    if ( (rc = iommu_domain_init(d, config->iommu_opts)) != 0 )
         goto fail;
 
     psr_domain_init(d);
xen/common/domain.c +7 -0
index 7dfb257c50..93bb0d4b51 100644
@@ -308,6 +308,13 @@ static int sanitise_domain_config(struct xen_domctl_createdomain *config)
         return -EINVAL;
     }
 
+    if ( !(config->flags & XEN_DOMCTL_CDF_iommu) && config->iommu_opts )
+    {
+        dprintk(XENLOG_INFO,
+                "IOMMU options specified but IOMMU not enabled\n");
+        return -EINVAL;
+    }
+
     if ( config->max_vcpus < 1 )
     {
         dprintk(XENLOG_INFO, "No vCPUS\n");
xen/common/domctl.c +0 -13
index 5dcfe3c8f6..6e6e9b9866 100644
@@ -515,19 +515,6 @@ long do_domctl(XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
             rover = dom;
         }
 
-        /*
-         * For now, make sure the createdomain IOMMU flag is set if the
-         * IOMMU is enabled. When the flag comes under toolstack control
-         * this can go away.
-         */
-        if ( op->u.createdomain.flags & XEN_DOMCTL_CDF_iommu )
-        {
-            ASSERT_UNREACHABLE();
-            return -EINVAL;
-        }
-        if ( iommu_enabled )
-            op->u.createdomain.flags |= XEN_DOMCTL_CDF_iommu;
-
         d = domain_create(dom, &op->u.createdomain, false);
         if ( IS_ERR(d) )
         {
xen/drivers/passthrough/iommu.c +12 -1
index aaf3b9fac0..cea770d2b6 100644
@@ -163,7 +163,7 @@ static void __hwdom_init check_hwdom_reqs(struct domain *d)
     iommu_hwdom_strict = true;
 }
 
-int iommu_domain_init(struct domain *d)
+int iommu_domain_init(struct domain *d, unsigned int opts)
 {
     struct domain_iommu *hd = dom_iommu(d);
     int ret = 0;
@@ -183,6 +183,15 @@ int iommu_domain_init(struct domain *d)
     if ( is_hardware_domain(d) )
         check_hwdom_reqs(d); /* may modify iommu_hwdom_strict */
 
+    /*
+     * Use shared page tables for HAP and IOMMU if the global option
+     * is enabled (from which we can infer the h/w is capable) and
+     * the domain options do not disallow it. HAP must, of course, also
+     * be enabled.
+     */
+    hd->hap_pt_share = hap_enabled(d) && iommu_hap_pt_share &&
+        !(opts & XEN_DOMCTL_IOMMU_no_sharept);
+
     /*
      * NB: 'relaxed' h/w domains don't need the IOMMU mappings to be kept
      *     in-sync with their assigned pages because all host RAM will be
@@ -191,6 +200,8 @@ int iommu_domain_init(struct domain *d)
     if ( !is_hardware_domain(d) || iommu_hwdom_strict )
         hd->need_sync = !iommu_use_hap_pt(d);
 
+    ASSERT(!(hd->need_sync && hd->hap_pt_share));
+
     return 0;
 }
 
xen/include/public/domctl.h +5 -1
index 5f55a2f6e1..0038fa6617 100644
@@ -38,7 +38,7 @@
 #include "hvm/save.h"
 #include "memory.h"
 
-#define XEN_DOMCTL_INTERFACE_VERSION 0x00000011
+#define XEN_DOMCTL_INTERFACE_VERSION 0x00000012
 
 /*
  * NB. xen_domctl.domain is an IN/OUT parameter for this operation.
@@ -70,6 +70,10 @@ struct xen_domctl_createdomain {
 
     uint32_t flags;
 
+#define _XEN_DOMCTL_IOMMU_no_sharept  0
+#define XEN_DOMCTL_IOMMU_no_sharept   (1U << _XEN_DOMCTL_IOMMU_no_sharept)
+    uint32_t iommu_opts;
+
     /*
      * Various domain limits, which impact the quantity of resources (global
      * mapping space, xenheap, etc) a guest may consume.
xen/include/xen/iommu.h +13 -6
index 87f9129b99..11c47f2151 100644
@@ -73,7 +73,7 @@ extern unsigned int iommu_dev_iotlb_timeout;
 int iommu_setup(void);
 int iommu_hardware_setup(void);
 
-int iommu_domain_init(struct domain *d);
+int iommu_domain_init(struct domain *d, unsigned int opts);
 void iommu_hwdom_init(struct domain *d);
 void iommu_domain_destroy(struct domain *d);
 
@@ -263,9 +263,17 @@ struct domain_iommu {
     DECLARE_BITMAP(features, IOMMU_FEAT_count);
 
     /*
-     * Does the guest reqire mappings to be synchonized, to maintain
-     * the default dfn == pfn map. (See comment on dfn at the top of
-     * include/xen/mm.h).
+     * Does the guest share HAP mapping with the IOMMU? This is always
+     * true for ARM systems and may be true for x86 systems where the
+     * the hardware is capable.
+     */
+    bool hap_pt_share;
+
+    /*
+     * Does the guest require mappings to be synchronized, to maintain
+     * the default dfn == pfn map? (See comment on dfn at the top of
+     * include/xen/mm.h). Note that hap_pt_share == false does not
+     * necessarily imply this is true.
      */
     bool need_sync;
 };
@@ -275,8 +283,7 @@ struct domain_iommu {
 #define iommu_clear_feature(d, f) clear_bit(f, dom_iommu(d)->features)
 
 /* Are we using the domain P2M table as its IOMMU pagetable? */
-#define iommu_use_hap_pt(d) \
-    (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share)
+#define iommu_use_hap_pt(d)       (dom_iommu(d)->hap_pt_share)
 
 /* Does the IOMMU pagetable need to be kept synchronized with the P2M */
 #ifdef CONFIG_HAS_PASSTHROUGH
Signature · 7 lines
2.20.1.2.gb21ebb671


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#23 lore raw

Re: [Xen-devel] [PATCH v8 6/6] introduce a 'passthrough' configuration option to xl.cfg...

Hi,

xen/include/xen/iommu.h struct domain_iommu { · line 269

Quoted 11 lines · Paul Durrant
xen/include/xen/iommu.h +4 -3
@@ -263,9 +263,17 @@ struct domain_iommu {
      DECLARE_BITMAP(features, IOMMU_FEAT_count);
 
      /*
-     * Does the guest reqire mappings to be synchonized, to maintain
-     * the default dfn == pfn map. (See comment on dfn at the top of
-     * include/xen/mm.h).
+     * Does the guest share HAP mapping with the IOMMU? This is always
+     * true for ARM systems and may be true for x86 systems where the
+     * the hardware is capable.
+     */

I am worried that such comment may rot over the time. For instance, if we either add a new architecture or decide to stop sharing PT on Arm.

I vaguely recall some potential issues with the MSI doorbells that would require us to unshare the PT when they will be supported in guest.

I would suggest to either refers to the implementation of iommu_use_hap_pt() or drop completely the second sentence.

Quoted 12 lines
xen/include/xen/iommu.h +7 -0
+    bool hap_pt_share;
+
+    /*
+     * Does the guest require mappings to be synchronized, to maintain
+     * the default dfn == pfn map? (See comment on dfn at the top of
+     * include/xen/mm.h). Note that hap_pt_share == false does not
+     * necessarily imply this is true.
       */
      bool need_sync;
  };

xen/include/xen/iommu.h struct domain_iommu { · line 289

Quoted 9 lines
xen/include/xen/iommu.h +1 -2
@@ -275,8 +283,7 @@ struct domain_iommu {
  #define iommu_clear_feature(d, f) clear_bit(f, dom_iommu(d)->features)
 
  /* Are we using the domain P2M table as its IOMMU pagetable? */
-#define iommu_use_hap_pt(d) \
-    (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share)
+#define iommu_use_hap_pt(d)       (dom_iommu(d)->hap_pt_share)
 
  /* Does the IOMMU pagetable need to be kept synchronized with the P2M */
  #ifdef CONFIG_HAS_PASSTHROUGH

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#24 lore raw

Re: [Xen-devel] [PATCH v8 6/6] introduce a 'passthrough' configuration option to xl.cfg...

Quoted 30 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 05 September 2019 20:59
To: Paul Durrant <[email protected]>; [email protected]
Cc: Jan Beulich <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Anthony Perard <[email protected]>; Christian Lindig <[email protected]>; David Scott <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [PATCH v8 6/6] introduce a 'passthrough' configuration option to xl.cfg...

Hi,

xen/include/xen/iommu.h struct domain_iommu { · line 269

Quoted 11 lines · Paul Durrant
xen/include/xen/iommu.h +4 -3
@@ -263,9 +263,17 @@ struct domain_iommu {
      DECLARE_BITMAP(features, IOMMU_FEAT_count);
 
      /*
-     * Does the guest reqire mappings to be synchonized, to maintain
-     * the default dfn == pfn map. (See comment on dfn at the top of
-     * include/xen/mm.h).
+     * Does the guest share HAP mapping with the IOMMU? This is always
+     * true for ARM systems and may be true for x86 systems where the
+     * the hardware is capable.
+     */

I am worried that such comment may rot over the time. For instance, if we either add a new architecture or decide to stop sharing PT on Arm.

Quoted 5 lines

I vaguely recall some potential issues with the MSI doorbells that would require us to unshare the PT when they will be supported in guest.

I would suggest to either refers to the implementation of iommu_use_hap_pt() or drop completely the second sentence.

Ok, I'll just drop the sentence.

Paul

Quoted 19 lines
Quoted 19 lines
xen/include/xen/iommu.h +8 -2
+    bool hap_pt_share;
+
+    /*
+     * Does the guest require mappings to be synchronized, to maintain
+     * the default dfn == pfn map? (See comment on dfn at the top of
+     * include/xen/mm.h). Note that hap_pt_share == false does not
+     * necessarily imply this is true.
       */
      bool need_sync;
  };
@@ -275,8 +283,7 @@ struct domain_iommu {
  #define iommu_clear_feature(d, f) clear_bit(f, dom_iommu(d)->features)
 
  /* Are we using the domain P2M table as its IOMMU pagetable? */
-#define iommu_use_hap_pt(d) \
-    (hap_enabled(d) && is_iommu_enabled(d) && iommu_hap_pt_share)
+#define iommu_use_hap_pt(d)       (dom_iommu(d)->hap_pt_share)
 
Quoted 8 lines

xen/include/xen/iommu.h struct domain_iommu { · line 289

Quoted 3 lines
xen/include/xen/iommu.h +0 -0
  /* Does the IOMMU pagetable need to be kept synchronized with the P2M */
  #ifdef CONFIG_HAS_PASSTHROUGH

Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#25 lore raw

Re: [Xen-devel] [PATCH v8 6/6] introduce a 'passthrough' configuration option to xl.cfg...

In reply to Paul Durrant

Hi,

Quoted 106 lines · Paul Durrant

...and hence the ability to disable IOMMU mappings, and control EPT sharing.

This patch introduces a new 'libxl_passthrough' enumeration into libxl_domain_create_info. The value will be set by xl either when it parses a new 'passthrough' option in xl.cfg, or implicitly if there is passthrough hardware specified for the domain.

If the value of the passthrough configuration option is 'disabled' then the XEN_DOMCTL_CDF_iommu flag will be clear in the xen_domctl_createdomain flags, thus allowing the toolstack to control whether the domain gets IOMMU mappings or not (where previously they were globally set).

If the value of the passthrough configuration option is 'sync_pt' then a new 'iommu_opts' field in xen_domctl_createdomain will be set with the value XEN_DOMCTL_IOMMU_no_sharept. This will override the global default set in iommu_hap_pt_share, thus allowing the toolstack to control whether EPT sharing is used for the domain.

NOTE: The 'iommu_memkb' overhead in libxl_domain_build_info will only be
set to zero if passthrough is 'disabled'. It is not safe to set the overhead to zero in the 'share_pt' case because the toolstack has no means of knowing whether the hardware actually supports IOMMU page table sharing.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Christian Lindig <[email protected]>
Cc: David Scott <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html

v7:
- Added missing breaks
- Added missing ocaml binding changes

v6:
- Remove the libxl_physinfo() call since it's usefulness is limited to x86

v5:
- Expand xen_domctl_createdomain flags field and hence bump interface version
- Fix spelling mistakes in context line

---
  docs/man/xl.cfg.5.pod.in            |  52 +++++++++++
  tools/libxl/libxl.h                 |   5 +
  tools/libxl/libxl_create.c          |   9 ++
  tools/libxl/libxl_types.idl         |   7 ++
  tools/ocaml/libs/xc/xenctrl.ml      |   4 +
  tools/ocaml/libs/xc/xenctrl.mli     |   4 +
  tools/ocaml/libs/xc/xenctrl_stubs.c |  15 ++-
  tools/xl/xl_parse.c                 | 140 ++++++++++++++++++----------
  xen/arch/arm/domain.c               |  10 +-
  xen/arch/x86/domain.c               |   2 +-
  xen/common/domain.c                 |   7 ++
  xen/common/domctl.c                 |  13 ---
  xen/drivers/passthrough/iommu.c     |  13 ++-
  xen/include/public/domctl.h         |   6 +-
  xen/include/xen/iommu.h             |  19 ++--
  15 files changed, 229 insertions(+), 77 deletions(-)
docs/man/xl.cfg.5.pod.in +25 -0
index c99d40307e..fd35685e9e 100644
@@ -605,6 +605,58 @@ option should only be used with a trusted device tree.
  Note that the partial device tree should avoid using the phandle 65000
  which is reserved by the toolstack.
 
+=item B<passthrough="STRING">
+
+Specify whether IOMMU mappings are enabled for the domain and hence whether
+it will be enabled for passthrough hardware. Valid values for this option
+are:
+
+=over 4
+
+=item B<disabled>
+
+IOMMU mappings are disabled for the domain and so hardware may not be
+passed through.
+
+This option is the default if no passthrough hardware is specified
+in the domain's configuration.
+
+=item B<sync_pt>
+
+This option means that IOMMU mappings will be synchronized with the
+domain's P2M table as follows:
+
+For a PV domain, all writable pages assigned to the domain are identity
+mapped by MFN in the IOMMU page table. Thus a device driver running in the
+domain may program passthrough hardware for DMA using MFN values
+(i.e. host/machine frame numbers) looked up in its P2M.

docs/man/xl.cfg.5.pod.in option should only be used with a trusted device tree. · line 642

Quoted 10 lines
docs/man/xl.cfg.5.pod.in +10 -0
+
+For an HVM domain, all non-foreign RAM pages present in its P2M will be
+mapped by GFN in the IOMMU page table. Thus a device driver running in the
+domain may program passthrough hardware using GFN values (i.e. guest
+physical frame numbers) without any further translation.
+
+This option is the default if the domain is PV and passthrough hardware
+is specified in the configuration.
+
+This option is not available on Arm.

I don't particularly like the idea to allow the user (or any external toolstack) to rely on passthrough=share_pt for Arm. This may put us in a corner if we ever support IOMMU that can't use the CPU PT (I know a few of them).

It feels to me we want a "default" that can let the toolstack (or the hypervisor) to select what ever is the most suitable for the preferred way.

For now default, could be aliased to share_pt for Arm in the toolstack. The point here is any toolstack built on top of libxl will not rely on passthrough=share_pt.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#26 lore raw

Re: [Xen-devel] [PATCH v8 6/6] introduce a 'passthrough' configuration option to xl.cfg...

Quoted 128 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 05 September 2019 21:28
To: Paul Durrant <[email protected]>; [email protected]
Cc: Jan Beulich <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Anthony Perard <[email protected]>; Christian Lindig <[email protected]>; David Scott <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [PATCH v8 6/6] introduce a 'passthrough' configuration option to xl.cfg...

Hi,

Quoted 113 lines · Paul Durrant

...and hence the ability to disable IOMMU mappings, and control EPT sharing.

This patch introduces a new 'libxl_passthrough' enumeration into libxl_domain_create_info. The value will be set by xl either when it parses a new 'passthrough' option in xl.cfg, or implicitly if there is passthrough hardware specified for the domain.

If the value of the passthrough configuration option is 'disabled' then the XEN_DOMCTL_CDF_iommu flag will be clear in the xen_domctl_createdomain flags, thus allowing the toolstack to control whether the domain gets IOMMU mappings or not (where previously they were globally set).

If the value of the passthrough configuration option is 'sync_pt' then a new 'iommu_opts' field in xen_domctl_createdomain will be set with the value XEN_DOMCTL_IOMMU_no_sharept. This will override the global default set in iommu_hap_pt_share, thus allowing the toolstack to control whether EPT sharing is used for the domain.

NOTE: The 'iommu_memkb' overhead in libxl_domain_build_info will only be
set to zero if passthrough is 'disabled'. It is not safe to set the overhead to zero in the 'share_pt' case because the toolstack has no means of knowing whether the hardware actually supports IOMMU page table sharing.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>
Cc: Anthony PERARD <[email protected]>
Cc: Christian Lindig <[email protected]>
Cc: David Scott <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html

v7:
- Added missing breaks
- Added missing ocaml binding changes

v6:
- Remove the libxl_physinfo() call since it's usefulness is limited to x86

v5:
- Expand xen_domctl_createdomain flags field and hence bump interface version
- Fix spelling mistakes in context line

---
  docs/man/xl.cfg.5.pod.in            |  52 +++++++++++
  tools/libxl/libxl.h                 |   5 +
  tools/libxl/libxl_create.c          |   9 ++
  tools/libxl/libxl_types.idl         |   7 ++
  tools/ocaml/libs/xc/xenctrl.ml      |   4 +
  tools/ocaml/libs/xc/xenctrl.mli     |   4 +
  tools/ocaml/libs/xc/xenctrl_stubs.c |  15 ++-
  tools/xl/xl_parse.c                 | 140 ++++++++++++++++++----------
  xen/arch/arm/domain.c               |  10 +-
  xen/arch/x86/domain.c               |   2 +-
  xen/common/domain.c                 |   7 ++
  xen/common/domctl.c                 |  13 ---
  xen/drivers/passthrough/iommu.c     |  13 ++-
  xen/include/public/domctl.h         |   6 +-
  xen/include/xen/iommu.h             |  19 ++--
  15 files changed, 229 insertions(+), 77 deletions(-)
docs/man/xl.cfg.5.pod.in +32 -0
index c99d40307e..fd35685e9e 100644
@@ -605,6 +605,58 @@ option should only be used with a trusted device tree.
  Note that the partial device tree should avoid using the phandle 65000
  which is reserved by the toolstack.
 
+=item B<passthrough="STRING">
+
+Specify whether IOMMU mappings are enabled for the domain and hence whether
+it will be enabled for passthrough hardware. Valid values for this option
+are:
+
+=over 4
+
+=item B<disabled>
+
+IOMMU mappings are disabled for the domain and so hardware may not be
+passed through.
+
+This option is the default if no passthrough hardware is specified
+in the domain's configuration.
+
+=item B<sync_pt>
+
+This option means that IOMMU mappings will be synchronized with the
+domain's P2M table as follows:
+
+For a PV domain, all writable pages assigned to the domain are identity
+mapped by MFN in the IOMMU page table. Thus a device driver running in the
+domain may program passthrough hardware for DMA using MFN values
+(i.e. host/machine frame numbers) looked up in its P2M.
+
+For an HVM domain, all non-foreign RAM pages present in its P2M will be
+mapped by GFN in the IOMMU page table. Thus a device driver running in the
+domain may program passthrough hardware using GFN values (i.e. guest
+physical frame numbers) without any further translation.
+
+This option is the default if the domain is PV and passthrough hardware
Quoted 8 lines

docs/man/xl.cfg.5.pod.in option should only be used with a trusted device tree. · line 642

Quoted 3 lines
docs/man/xl.cfg.5.pod.in +3 -0
+is specified in the configuration.
+
+This option is not available on Arm.

I don't particularly like the idea to allow the user (or any external toolstack) to rely on passthrough=share_pt for Arm. This may put us in a corner if we ever support IOMMU that can't use the CPU PT (I know a few of them).

I could just say 'not currently available' and, if ARM gains a non-shared implementation then the manpage could be changed. I don't think xl.cfg files need to be considered a stable interface, do they?

Quoted 7 lines

It feels to me we want a "default" that can let the toolstack (or the hypervisor) to select what ever is the most suitable for the preferred way.

For now default, could be aliased to share_pt for Arm in the toolstack. The point here is any toolstack built on top of libxl will not rely on passthrough=share_pt.

I don't really like that. The 'default' option is chosen by not putting any option in the cfg, I don't see why it needs to be explicit for this option.

Paul

Quoted 5 lines

Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#27 lore raw

Re: [Xen-devel] [PATCH v8 6/6] introduce a 'passthrough' configuration option to xl.cfg...

Hi Paul,

Quoted 137 lines · Paul Durrant
Quoted 128 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 05 September 2019 21:28
To: Paul Durrant <[email protected]>; [email protected]
Cc: Jan Beulich <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Anthony Perard <[email protected]>; Christian Lindig <[email protected]>; David Scott <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [PATCH v8 6/6] introduce a 'passthrough' configuration option to xl.cfg...

Hi,

Quoted 113 lines · Paul Durrant

...and hence the ability to disable IOMMU mappings, and control EPT sharing.

This patch introduces a new 'libxl_passthrough' enumeration into libxl_domain_create_info. The value will be set by xl either when it parses a new 'passthrough' option in xl.cfg, or implicitly if there is passthrough hardware specified for the domain.

If the value of the passthrough configuration option is 'disabled' then the XEN_DOMCTL_CDF_iommu flag will be clear in the xen_domctl_createdomain flags, thus allowing the toolstack to control whether the domain gets IOMMU mappings or not (where previously they were globally set).

If the value of the passthrough configuration option is 'sync_pt' then a new 'iommu_opts' field in xen_domctl_createdomain will be set with the value XEN_DOMCTL_IOMMU_no_sharept. This will override the global default set in iommu_hap_pt_share, thus allowing the toolstack to control whether EPT sharing is used for the domain.

NOTE: The 'iommu_memkb' overhead in libxl_domain_build_info will only be
set to zero if passthrough is 'disabled'. It is not safe to set the overhead to zero in the 'share_pt' case because the toolstack has no means of knowing whether the hardware actually supports IOMMU page table sharing.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>,
Cc: Anthony PERARD <[email protected]>
Cc: Christian Lindig <[email protected]>
Cc: David Scott <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg02267.html

v7:
- Added missing breaks
- Added missing ocaml binding changes

v6:
- Remove the libxl_physinfo() call since it's usefulness is limited to x86

v5:
- Expand xen_domctl_createdomain flags field and hence bump interface version
- Fix spelling mistakes in context line

---
   docs/man/xl.cfg.5.pod.in            |  52 +++++++++++
   tools/libxl/libxl.h                 |   5 +
   tools/libxl/libxl_create.c          |   9 ++
   tools/libxl/libxl_types.idl         |   7 ++
   tools/ocaml/libs/xc/xenctrl.ml      |   4 +
   tools/ocaml/libs/xc/xenctrl.mli     |   4 +
   tools/ocaml/libs/xc/xenctrl_stubs.c |  15 ++-
   tools/xl/xl_parse.c                 | 140 ++++++++++++++++++----------
   xen/arch/arm/domain.c               |  10 +-
   xen/arch/x86/domain.c               |   2 +-
   xen/common/domain.c                 |   7 ++
   xen/common/domctl.c                 |  13 ---
   xen/drivers/passthrough/iommu.c     |  13 ++-
   xen/include/public/domctl.h         |   6 +-
   xen/include/xen/iommu.h             |  19 ++--
   15 files changed, 229 insertions(+), 77 deletions(-)
docs/man/xl.cfg.5.pod.in +32 -0
index c99d40307e..fd35685e9e 100644
@@ -605,6 +605,58 @@ option should only be used with a trusted device tree.
   Note that the partial device tree should avoid using the phandle 65000
   which is reserved by the toolstack.
 
+=item B<passthrough="STRING">
+
+Specify whether IOMMU mappings are enabled for the domain and hence whether
+it will be enabled for passthrough hardware. Valid values for this option
+are:
+
+=over 4
+
+=item B<disabled>
+
+IOMMU mappings are disabled for the domain and so hardware may not be
+passed through.
+
+This option is the default if no passthrough hardware is specified
+in the domain's configuration.
+
+=item B<sync_pt>
+
+This option means that IOMMU mappings will be synchronized with the
+domain's P2M table as follows:
+
+For a PV domain, all writable pages assigned to the domain are identity
+mapped by MFN in the IOMMU page table. Thus a device driver running in the
+domain may program passthrough hardware for DMA using MFN values
+(i.e. host/machine frame numbers) looked up in its P2M.
+
+For an HVM domain, all non-foreign RAM pages present in its P2M will be
+mapped by GFN in the IOMMU page table. Thus a device driver running in the
+domain may program passthrough hardware using GFN values (i.e. guest
+physical frame numbers) without any further translation.
+
+This option is the default if the domain is PV and passthrough hardware
Quoted 8 lines

docs/man/xl.cfg.5.pod.in option should only be used with a trusted device tree. · line 642

Quoted 3 lines
docs/man/xl.cfg.5.pod.in +3 -0
+is specified in the configuration.
+
+This option is not available on Arm.

I don't particularly like the idea to allow the user (or any external toolstack) to rely on passthrough=share_pt for Arm. This may put us in a corner if we ever support IOMMU that can't use the CPU PT (I know a few of them).

Quoted 1 line

I could just say 'not currently available' and, if ARM gains a non-shared implementation then the manpage could be changed. I don't think xl.cfg files need to be considered a stable interface, do they?

I am not too worried about the xl.cfg files. My worry is regarding the libxl_types.idl which is definitely stable.

If you say the field is currently not available for Arm, the field will still be fill up by a given value. That given value will have to be handled differently when ARM gains a non-shared implementation.

Quoted 9 lines
Quoted 7 lines

It feels to me we want a "default" that can let the toolstack (or the hypervisor) to select what ever is the most suitable for the preferred way.

For now default, could be aliased to share_pt for Arm in the toolstack. The point here is any toolstack built on top of libxl will not rely on passthrough=share_pt.

I don't really like that. The 'default' option is chosen by not putting any option in the cfg, I don't see why it needs to be explicit for this option.r

Well, here you impose the user to know how to configure the IOMMU (i.e. shared vs non-shared). He/She may not know about it and therefore he/she will have to try different value until Xen accepts it.

For the benefits of the user (and make it easy to extend in the future) having an option to say "let Xen chose" allow an external toolstack to not replicate the exact code you wrote in xl_parse.c.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#28 lore raw

Re: [Xen-devel] [PATCH v8 6/6] introduce a 'passthrough' configuration option to xl.cfg...

Quoted 158 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 06 September 2019 10:06
To: Paul Durrant <[email protected]>; [email protected]
Cc: Jan Beulich <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Anthony Perard <[email protected]>; Christian Lindig <[email protected]>; David Scott <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [PATCH v8 6/6] introduce a 'passthrough' configuration option to xl.cfg...

Hi Paul,

Quoted 6 lines · Paul Durrant
Quoted 6 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 05 September 2019 21:28
To: Paul Durrant <[email protected]>; [email protected]
Cc: Jan Beulich <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Konrad

Rzeszutek

Quoted 51 lines
Quoted 51 lines

Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Anthony Perard <[email protected]>; Christian Lindig <[email protected]>; David Scott <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [PATCH v8 6/6] introduce a 'passthrough' configuration option to xl.cfg...

Hi,

Quoted 42 lines · Paul Durrant

...and hence the ability to disable IOMMU mappings, and control EPT sharing.

This patch introduces a new 'libxl_passthrough' enumeration into libxl_domain_create_info. The value will be set by xl either when it parses a new 'passthrough' option in xl.cfg, or implicitly if there is passthrough hardware specified for the domain.

If the value of the passthrough configuration option is 'disabled' then the XEN_DOMCTL_CDF_iommu flag will be clear in the xen_domctl_createdomain flags, thus allowing the toolstack to control whether the domain gets IOMMU mappings or not (where previously they were globally set).

If the value of the passthrough configuration option is 'sync_pt' then a new 'iommu_opts' field in xen_domctl_createdomain will be set with the value XEN_DOMCTL_IOMMU_no_sharept. This will override the global default set in iommu_hap_pt_share, thus allowing the toolstack to control whether EPT sharing is used for the domain.

NOTE: The 'iommu_memkb' overhead in libxl_domain_build_info will only be
set to zero if passthrough is 'disabled'. It is not safe to set the overhead to zero in the 'share_pt' case because the toolstack has no means of knowing whether the hardware actually supports IOMMU page table sharing.

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Tim Deegan <[email protected]>,
Cc: Anthony PERARD <[email protected]>
Cc: Christian Lindig <[email protected]>
Cc: David Scott <[email protected]>
Cc: Volodymyr Babchuk <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

Quoted 1 line
Quoted 1 line
Quoted 1 line

Previously part of series https://lists.xenproject.org/archives/html/xen-devel/2019-

07/msg02267.html

Quoted 79 lines
Quoted 70 lines
Quoted 70 lines

v7:
- Added missing breaks
- Added missing ocaml binding changes

v6:
- Remove the libxl_physinfo() call since it's usefulness is limited to x86

v5:
- Expand xen_domctl_createdomain flags field and hence bump interface version
- Fix spelling mistakes in context line

---
   docs/man/xl.cfg.5.pod.in            |  52 +++++++++++
   tools/libxl/libxl.h                 |   5 +
   tools/libxl/libxl_create.c          |   9 ++
   tools/libxl/libxl_types.idl         |   7 ++
   tools/ocaml/libs/xc/xenctrl.ml      |   4 +
   tools/ocaml/libs/xc/xenctrl.mli     |   4 +
   tools/ocaml/libs/xc/xenctrl_stubs.c |  15 ++-
   tools/xl/xl_parse.c                 | 140 ++++++++++++++++++----------
   xen/arch/arm/domain.c               |  10 +-
   xen/arch/x86/domain.c               |   2 +-
   xen/common/domain.c                 |   7 ++
   xen/common/domctl.c                 |  13 ---
   xen/drivers/passthrough/iommu.c     |  13 ++-
   xen/include/public/domctl.h         |   6 +-
   xen/include/xen/iommu.h             |  19 ++--
   15 files changed, 229 insertions(+), 77 deletions(-)
docs/man/xl.cfg.5.pod.in +32 -0
index c99d40307e..fd35685e9e 100644
@@ -605,6 +605,58 @@ option should only be used with a trusted device tree.
   Note that the partial device tree should avoid using the phandle 65000
   which is reserved by the toolstack.
 
+=item B<passthrough="STRING">
+
+Specify whether IOMMU mappings are enabled for the domain and hence whether
+it will be enabled for passthrough hardware. Valid values for this option
+are:
+
+=over 4
+
+=item B<disabled>
+
+IOMMU mappings are disabled for the domain and so hardware may not be
+passed through.
+
+This option is the default if no passthrough hardware is specified
+in the domain's configuration.
+
+=item B<sync_pt>
+
+This option means that IOMMU mappings will be synchronized with the
+domain's P2M table as follows:
+
+For a PV domain, all writable pages assigned to the domain are identity
+mapped by MFN in the IOMMU page table. Thus a device driver running in the
+domain may program passthrough hardware for DMA using MFN values
+(i.e. host/machine frame numbers) looked up in its P2M.
+
+For an HVM domain, all non-foreign RAM pages present in its P2M will be
+mapped by GFN in the IOMMU page table. Thus a device driver running in the
+domain may program passthrough hardware using GFN values (i.e. guest
+physical frame numbers) without any further translation.
+
+This option is the default if the domain is PV and passthrough hardware
Quoted 8 lines

docs/man/xl.cfg.5.pod.in option should only be used with a trusted device tree. · line 642

Quoted 3 lines
docs/man/xl.cfg.5.pod.in +3 -0
+is specified in the configuration.
+
+This option is not available on Arm.

I don't particularly like the idea to allow the user (or any external toolstack) to rely on passthrough=share_pt for Arm. This may put us in a corner if we ever support IOMMU that can't use the CPU PT (I know a few of them).

Quoted 1 line

I could just say 'not currently available' and, if ARM gains a non-shared implementation then the

manpage could be changed. I don't think xl.cfg files need to be considered a stable interface, do they?

Quoted 7 lines

I am not too worried about the xl.cfg files. My worry is regarding the libxl_types.idl which is definitely stable.

If you say the field is currently not available for Arm, the field will still be fill up by a given value. That given value will have to be handled differently when ARM gains a non-shared implementation.

The idl is not architecture specific so I don't see any issue there. If ARM gains a non-shared implementation then the toolstack would not need to change, just the documentation. Setting sync_pt results in the 'no shared PT' IOMMU option being passed to domain create. That would currently be rejected by ARM's sanitise_domain_config() but that, of course, can be easily changed.

Quoted 11 lines
Quoted 9 lines
Quoted 7 lines

It feels to me we want a "default" that can let the toolstack (or the hypervisor) to select what ever is the most suitable for the preferred way.

For now default, could be aliased to share_pt for Arm in the toolstack. The point here is any toolstack built on top of libxl will not rely on passthrough=share_pt.

I don't really like that. The 'default' option is chosen by not putting any option in the cfg, I

don't see why it needs to be explicit for this option.r

Quoted 7 lines

Well, here you impose the user to know how to configure the IOMMU (i.e. shared vs non-shared). He/She may not know about it and therefore he/she will have to try different value until Xen accepts it.

For the benefits of the user (and make it easy to extend in the future) having an option to say "let Xen chose" allow an external toolstack to not replicate the exact code you wrote in xl_parse.c.

Ok. I was discussing the Roger and Andy on IRC and I think I'm persuaded.

Paul

Quoted 5 lines

Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel