Skip to contentSkip to conversation

iommu/arm: Add Renesas IPMMU-VMSA support + Linux's iommu_fwspec

PATCH Superseded Oleksandr Tyshchenko 7 participants 59 messages Aug 2, 2019 – Aug 15, 2019

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v2 cover iommu/arm: Add Renesas IPMMU-VMSA support + Linux's iommu_fwspec

From: Oleksandr Tyshchenko <[email protected]>

The purpose of this patch series is to add IPMMU-VMSA support to Xen on ARM.

Besides new IOMMU driver, this series contains "iommu_fwspec" support and new API iommu_add_dt_device() for adding DT device to IOMMU.

The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.

Please note, this driver is supposed to work only with newest Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).

----------
This driver is based on Linux's IPMMU-VMSA driver from Renesas BSP:
https://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git/tree/drivers/iommu/ipmmu-vmsa.c?h=v4.14.75-ltsi/rcar-3.9.6
and Xen's SMMU driver:
xen/drivers/passthrough/arm/smmu.c

Although Xen driver has a lot in common with Linux driver, it is not a "direct ported" copy and should be treated as such.

The major differences compare to the Linux driver are:

1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.

So, Xen driver is supposed to be used with newest Gen3 SoC revisions only (H3 ES3.0, M3 ES3.0, etc.) which IPMMU H/W supports stage 2 translation table format.

2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.

3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.

----------
Driver was tested on Gen3 H3 ES3.0 based boards using current staging (7d1460c xen/arm: optee: fix compilation with GCC 4.8)
in a system with several DMA masters being assigned to different guest domains.

You can find it here:
repo: https://github.com/otyshchenko1/xen.git branch: ipmmu_upstream2

You can find previous discussions here:
[V1] https://lists.xenproject.org/archives/html/xen-devel/2019-06/msg01755.html

Oleksandr Tyshchenko (6):
iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff iommu/arm: Add ability to handle deferred probing request [RFC] xen/common: Introduce _xrealloc function iommu/arm: Add lightweight iommu_fwspec support iommu/arm: Introduce iommu_add_dt_device API
iommu/arm: Add Renesas IPMMU-VMSA support

 xen/arch/arm/domain_build.c                 |   12 +
 xen/arch/arm/platforms/Kconfig              |    1 +
 xen/common/device_tree.c                    |    1 +
 xen/common/xmalloc_tlsf.c                   |   21 +
 xen/drivers/passthrough/Kconfig             |   13 +
 xen/drivers/passthrough/arm/Makefile        |    3 +-
 xen/drivers/passthrough/arm/iommu.c         |   80 +-
 xen/drivers/passthrough/arm/iommu_fwspec.c  |   91 ++
 xen/drivers/passthrough/arm/iommu_helpers.c |   78 ++
 xen/drivers/passthrough/arm/ipmmu-vmsa.c    | 1342 +++++++++++++++++++++++++++
 xen/drivers/passthrough/arm/smmu.c          |   48 +-
 xen/include/asm-arm/device.h                |    7 +-
 xen/include/asm-arm/iommu.h                 |   12 +
 xen/include/asm-arm/iommu_fwspec.h          |   65 ++
 xen/include/xen/device_tree.h               |    1 +
 xen/include/xen/xmalloc.h                   |    1 +
 16 files changed, 1727 insertions(+), 49 deletions(-)
 create mode 100644 xen/drivers/passthrough/arm/iommu_fwspec.c
 create mode 100644 xen/drivers/passthrough/arm/iommu_helpers.c
 create mode 100644 xen/drivers/passthrough/arm/ipmmu-vmsa.c
 create mode 100644 xen/include/asm-arm/iommu_fwspec.h
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH v2 1/6 iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff

From: Oleksandr Tyshchenko <[email protected]>

Introduce a separate file to keep various helpers which could be used by more than one IOMMU driver in order not to duplicate code.

The first condidates to be moved to the new file are SMMU driver's "map_page/unmap_page" callbacks. There callbacks neither contain any SMMU specific info nor perform any SMMU specific actions and are going to be the same across all IOMMU drivers which H/W IP shares P2M with the CPU like SMMU does.

So, move callbacks to iommu_helpers.c for the upcoming IPMMU driver to be able to re-use them.

---
 xen/drivers/passthrough/arm/Makefile        |  2 +-
 xen/drivers/passthrough/arm/iommu_helpers.c | 78 +++++++++++++++++++++++++++++
 xen/drivers/passthrough/arm/smmu.c          | 48 +-----------------
 xen/include/asm-arm/iommu.h                 |  7 +++
 4 files changed, 88 insertions(+), 47 deletions(-)
 create mode 100644 xen/drivers/passthrough/arm/iommu_helpers.c
xen/drivers/passthrough/arm/Makefile +1 -1
index b3efcfd..4abb87a 100644
@@ -1,2 +1,2 @@
-obj-y += iommu.o
+obj-y += iommu.o iommu_helpers.o
 obj-$(CONFIG_ARM_SMMU) += smmu.o
xen/drivers/passthrough/arm/iommu_helpers.c new file +78 -0

78 lines not on this page · raw message

xen/drivers/passthrough/arm/smmu.c +2 -46

60 lines not on this page · raw message

xen/include/asm-arm/iommu.h +7 -0
index 904c9ae..20d865e 100644
@@ -26,6 +26,13 @@ struct arch_iommu
 const struct iommu_ops *iommu_get_ops(void);
 void iommu_set_ops(const struct iommu_ops *ops);
 
+/* mapping helpers */
+int __must_check arm_iommu_map_page(struct domain *d, dfn_t dfn, mfn_t mfn,
+                                    unsigned int flags,
+                                    unsigned int *flush_flags);
+int __must_check arm_iommu_unmap_page(struct domain *d, dfn_t dfn,
+                                      unsigned int *flush_flags);
+
 #endif /* __ARCH_ARM_IOMMU_H__ */
 
 /*
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

Re: [Xen-devel] [PATCH V2 1/6] iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff

Hi Oleksandr,

Quoted 6 lines · Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

Introduce a separate file to keep various helpers which could be used by more than one IOMMU driver in order not to duplicate code.

The first condidates to be moved to the new file are SMMU driver's

NIT: s/condidates/candidates/

Quoted 31 lines

"map_page/unmap_page" callbacks. There callbacks neither contain any SMMU specific info nor perform any SMMU specific actions and are going to be the same across all IOMMU drivers which H/W IP shares P2M with the CPU like SMMU does.

So, move callbacks to iommu_helpers.c for the upcoming IPMMU driver to be able to re-use them.

---
  xen/drivers/passthrough/arm/Makefile        |  2 +-
  xen/drivers/passthrough/arm/iommu_helpers.c | 78 +++++++++++++++++++++++++++++
  xen/drivers/passthrough/arm/smmu.c          | 48 +-----------------
  xen/include/asm-arm/iommu.h                 |  7 +++
  4 files changed, 88 insertions(+), 47 deletions(-)

create mode 100644 xen/drivers/passthrough/arm/iommu_helpers.c

xen/drivers/passthrough/arm/Makefile +1 -1
index b3efcfd..4abb87a 100644
@@ -1,2 +1,2 @@
-obj-y += iommu.o
+obj-y += iommu.o iommu_helpers.o
  obj-$(CONFIG_ARM_SMMU) += smmu.o
xen/drivers/passthrough/arm/iommu_helpers.c new file +0 -0
new file mode 100644
index 0000000..53e8daa

xen/drivers/passthrough/arm/iommu_helpers.c line 6

Quoted 6 lines
xen/drivers/passthrough/arm/iommu_helpers.c new file +6 -0
@@ -0,0 +1,78 @@
+/*
+ * xen/drivers/passthrough/arm/iommu_helpers.c
+ *
+ * Contains various helpers to be used by IOMMU drivers.
+ *
+ * Copyright (C) 2019 EPAM Systems Inc.

You mostly moved the code from the SMMU code, so the copyright there should be retain. As this is Xen modification the copyright here should be:

* Copyright (C) 2014 Linaro Limited.

Quoted 7 lines
xen/drivers/passthrough/arm/iommu_helpers.c new file +7 -0
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of

xen/drivers/passthrough/arm/iommu_helpers.c line 23

Quoted 10 lines
xen/drivers/passthrough/arm/iommu_helpers.c new file +10 -0
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/lib.h>
+#include <xen/sched.h>
+#include <xen/iommu.h>

Could you order the headers above alphabetical please?

And also as a extra NIT a newline before 'xen' headers and 'asm' one :).

xen/drivers/passthrough/arm/iommu_helpers.c line 24

Quoted 1 line
xen/drivers/passthrough/arm/iommu_helpers.c new file +1 -0
+#include <asm/device.h>

The rest of the good looks good to me.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#3 lore raw

Re: [Xen-devel] [PATCH V2 1/6] iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff

Hi, Julien

Quoted 10 lines · Oleksandr Tyshchenko
Quoted 6 lines

From: Oleksandr Tyshchenko <[email protected]>

Introduce a separate file to keep various helpers which could be used by more than one IOMMU driver in order not to duplicate code.

The first condidates to be moved to the new file are SMMU driver's

NIT: s/condidates/candidates/

ok

Quoted 42 lines
Quoted 33 lines

"map_page/unmap_page" callbacks. There callbacks neither contain any SMMU specific info nor perform any SMMU specific actions and are going to be the same across all IOMMU drivers which H/W IP shares P2M with the CPU like SMMU does.

So, move callbacks to iommu_helpers.c for the upcoming IPMMU driver to be able to re-use them.

---
  xen/drivers/passthrough/arm/Makefile        |  2 +- xen/drivers/passthrough/arm/iommu_helpers.c | 78
+++++++++++++++++++++++++++++
  xen/drivers/passthrough/arm/smmu.c          | 48 +----------------- xen/include/asm-arm/iommu.h                 |  7 +++

  4 files changed, 88 insertions(+), 47 deletions(-)

  create mode 100644 xen/drivers/passthrough/arm/iommu_helpers.c

xen/drivers/passthrough/arm/Makefile +1 -1
index b3efcfd..4abb87a 100644
@@ -1,2 +1,2 @@
-obj-y += iommu.o
+obj-y += iommu.o iommu_helpers.o

  obj-$(CONFIG_ARM_SMMU) += smmu.o

xen/drivers/passthrough/arm/iommu_helpers.c new file +0 -0
new file mode 100644
index 0000000..53e8daa

xen/drivers/passthrough/arm/iommu_helpers.c line 6

Quoted 6 lines
xen/drivers/passthrough/arm/iommu_helpers.c new file +6 -0
@@ -0,0 +1,78 @@
+/*
+ * xen/drivers/passthrough/arm/iommu_helpers.c
+ *
+ * Contains various helpers to be used by IOMMU drivers.
+ *
+ * Copyright (C) 2019 EPAM Systems Inc.

You mostly moved the code from the SMMU code, so the copyright there should be retain. As this is Xen modification the copyright here should be:

Quoted 1 line

 * Copyright (C) 2014 Linaro Limited.

Oh, yes. Sorry, forgot about it. Will add.

Quoted 22 lines
Quoted 15 lines
xen/drivers/passthrough/arm/iommu_helpers.c new file +12 -0
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; If not, see 

<http://www.gnu.org/licenses/>.
+ */

xen/drivers/passthrough/arm/iommu_helpers.c line 23

Quoted 4 lines
xen/drivers/passthrough/arm/iommu_helpers.c new file +4 -0
+
+#include <xen/lib.h>
+#include <xen/sched.h>
+#include <xen/iommu.h>

Could you order the headers above alphabetical please?

Quoted 1 line

And also as a extra NIT a newline before 'xen' headers and 'asm' one :).

Yes, will follow this rule.

Quoted 7 lines
Quoted 2 lines

+#include <asm/device.h>

The rest of the good looks good to me.

Cheers,

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#4 lore raw

PATCH v2 2/6 iommu/arm: Add ability to handle deferred probing request

In reply to Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.

In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen -EAGAIN to be used for indicating that device probing is deferred.

This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.

---
 xen/common/device_tree.c            |  1 +
 xen/drivers/passthrough/arm/iommu.c | 35 ++++++++++++++++++++++++++++++++++-
 xen/include/asm-arm/device.h        |  6 +++++-
 xen/include/xen/device_tree.h       |  1 +
 4 files changed, 41 insertions(+), 2 deletions(-)
xen/common/device_tree.c +1 -0
index e107c6f..6f37448 100644
@@ -1774,6 +1774,7 @@ static unsigned long __init unflatten_dt_node(const void *fdt,
         /* By default the device is not protected */
         np->is_protected = false;
         INIT_LIST_HEAD(&np->domain_list);
+        INIT_LIST_HEAD(&np->deferred_probe);
 
         if ( new_format )
         {
xen/drivers/passthrough/arm/iommu.c +34 -1
index 2135233..3195919 100644
@@ -20,6 +20,12 @@
 #include <xen/device_tree.h>
 #include <asm/device.h>
 
+/*
+ * Used to keep track of devices for which driver requested deferred probing
+ * (returns -EAGAIN).
+ */
+static LIST_HEAD(deferred_probe_list);
+
 static const struct iommu_ops *iommu_ops;
 
 const struct iommu_ops *iommu_get_ops(void)
@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops)
 
 int __init iommu_hardware_setup(void)
 {
-    struct dt_device_node *np;
+    struct dt_device_node *np, *tmp;
     int rc;
     unsigned int num_iommus = 0;
 
@@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void)
         rc = device_init(np, DEVICE_IOMMU, NULL);
         if ( !rc )
             num_iommus++;
+        else if (rc == -EAGAIN)
+            /*
+             * Driver requested deferred probing, so add this device to
+             * the deferred list for further processing.
+             */
+            list_add(&np->deferred_probe, &deferred_probe_list);
+    }
+
+    /*
+     * Process devices in the deferred list if at least one successfully
+     * probed device is present.
+     */
+    while ( !list_empty(&deferred_probe_list) && num_iommus )
+    {
+        list_for_each_entry_safe ( np, tmp, &deferred_probe_list,
+                                   deferred_probe )
+        {
+            rc = device_init(np, DEVICE_IOMMU, NULL);
+            if ( !rc )
+                num_iommus++;
+            if ( rc != -EAGAIN )
+                /*
+                 * Driver didn't request deferred probing, so remove this device
+                 * from the deferred list.
+                 */
+                list_del_init(&np->deferred_probe);
+        }
     }
 
     return ( num_iommus > 0 ) ? 0 : -ENODEV;
xen/include/asm-arm/device.h +5 -1
index 63a0f36..ee1c3bc 100644
@@ -44,7 +44,11 @@ struct device_desc {
     enum device_class class;
     /* List of devices supported by this driver */
     const struct dt_device_match *dt_match;
-    /* Device initialization */
+    /*
+     * Device initialization.
+     *
+     * -EAGAIN is used to indicate that device probing is deferred.
+     */
     int (*init)(struct dt_device_node *dev, const void *data);
 };
 
xen/include/xen/device_tree.h +1 -0
index 8315629..71b0e47 100644
@@ -93,6 +93,7 @@ struct dt_device_node {
     /* IOMMU specific fields */
     bool is_protected;
     struct list_head domain_list;
+    struct list_head deferred_probe;
 
     struct device dev;
 };
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#5 lore raw

Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request

Hi Oleksandr,

Quoted 32 lines · Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.

In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen -EAGAIN to be used for indicating that device probing is deferred.

This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.

---
  xen/common/device_tree.c            |  1 +
  xen/drivers/passthrough/arm/iommu.c | 35 ++++++++++++++++++++++++++++++++++-
  xen/include/asm-arm/device.h        |  6 +++++-
  xen/include/xen/device_tree.h       |  1 +
  4 files changed, 41 insertions(+), 2 deletions(-)
xen/common/device_tree.c +0 -0
index e107c6f..6f37448 100644

xen/common/device_tree.c static unsigned long __init unflatten_dt_node(const void *fdt, · line 1777

Quoted 4 lines
xen/common/device_tree.c +1 -0
@@ -1774,6 +1774,7 @@ static unsigned long __init unflatten_dt_node(const void *fdt,
          /* By default the device is not protected */
          np->is_protected = false;
          INIT_LIST_HEAD(&np->domain_list);
+        INIT_LIST_HEAD(&np->deferred_probe);

I am not entirely happy to add a new list_head field per node just for the benefits of boot code. Could we re-use domain_list (with a comment in the code and appropriate ASSERT)?

Quoted 8 lines
xen/common/device_tree.c +0 -0
          if ( new_format )
          {
xen/drivers/passthrough/arm/iommu.c +0 -0
index 2135233..3195919 100644

xen/drivers/passthrough/arm/iommu.c line 27

Quoted 8 lines
xen/drivers/passthrough/arm/iommu.c +5 -0
@@ -20,6 +20,12 @@
  #include <xen/device_tree.h>
  #include <asm/device.h>
 
+/*
+ * Used to keep track of devices for which driver requested deferred probing
+ * (returns -EAGAIN).
+ */
+static LIST_HEAD(deferred_probe_list);

This wants to be in init section as this is only used at boot.

Quoted 19 lines
xen/drivers/passthrough/arm/iommu.c +4 -1
+
  static const struct iommu_ops *iommu_ops;
 
  const struct iommu_ops *iommu_get_ops(void)
@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops)
 
  int __init iommu_hardware_setup(void)
  {
-    struct dt_device_node *np;
+    struct dt_device_node *np, *tmp;
      int rc;
      unsigned int num_iommus = 0;
 
@@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void)
          rc = device_init(np, DEVICE_IOMMU, NULL);
          if ( !rc )
              num_iommus++;
+        else if (rc == -EAGAIN)
+            /*

xen/drivers/passthrough/arm/iommu.c int __init iommu_hardware_setup(void) · line 71

Quoted 10 lines
xen/drivers/passthrough/arm/iommu.c +10 -0
+             * Driver requested deferred probing, so add this device to
+             * the deferred list for further processing.
+             */
+            list_add(&np->deferred_probe, &deferred_probe_list);
+    }
+
+    /*
+     * Process devices in the deferred list if at least one successfully
+     * probed device is present.
+     */

I think this can turn into an infinite loop if all device in deferred_probe_list still return -EDEFER_PROBE and num_iommus is a non-zero.

A better condition would be to check that at least one IOMMU is added at each loop. If not, then we should bail with an error because it likely means something is buggy.

Quoted 41 lines
xen/drivers/passthrough/arm/iommu.c +15 -0
+    while ( !list_empty(&deferred_probe_list) && num_iommus )
+    {
+        list_for_each_entry_safe ( np, tmp, &deferred_probe_list,
+                                   deferred_probe )
+        {
+            rc = device_init(np, DEVICE_IOMMU, NULL);
+            if ( !rc )
+                num_iommus++;
+            if ( rc != -EAGAIN )
+                /*
+                 * Driver didn't request deferred probing, so remove this device
+                 * from the deferred list.
+                 */
+                list_del_init(&np->deferred_probe);
+        }
      }
 
      return ( num_iommus > 0 ) ? 0 : -ENODEV;
xen/include/asm-arm/device.h +5 -1
index 63a0f36..ee1c3bc 100644
@@ -44,7 +44,11 @@ struct device_desc {
      enum device_class class;
      /* List of devices supported by this driver */
      const struct dt_device_match *dt_match;
-    /* Device initialization */
+    /*
+     * Device initialization.
+     *
+     * -EAGAIN is used to indicate that device probing is deferred.
+     */
      int (*init)(struct dt_device_node *dev, const void *data);
  };
 
xen/include/xen/device_tree.h +0 -0
index 8315629..71b0e47 100644

xen/include/xen/device_tree.h struct dt_device_node { · line 99

Quoted 7 lines
xen/include/xen/device_tree.h +1 -0
@@ -93,6 +93,7 @@ struct dt_device_node {
      /* IOMMU specific fields */
      bool is_protected;
      struct list_head domain_list;
+    struct list_head deferred_probe;
 
      struct device dev;
  };

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#6 lore raw

Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request

Quoted 1 line · Julien Grall

Hi Oleksandr,

Hi, Julien

Quoted 30 lines · Oleksandr Tyshchenko
Quoted 28 lines

From: Oleksandr Tyshchenko <[email protected]>

This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.

In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen -EAGAIN to be used for indicating that device probing is deferred.

This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.

---
  xen/common/device_tree.c            |  1 +
  xen/drivers/passthrough/arm/iommu.c | 35
++++++++++++++++++++++++++++++++++-
  xen/include/asm-arm/device.h        |  6 +++++-
  xen/include/xen/device_tree.h       |  1 +

  4 files changed, 41 insertions(+), 2 deletions(-)
Quoted 12 lines

xen/common/device_tree.c static unsigned long __init · line 1778

Quoted 10 lines
xen/common/device_tree.c +1 -0
index e107c6f..6f37448 100644
@@ -1774,6 +1774,7 @@ static unsigned long __init 
 unflatten_dt_node(const void *fdt,
           /* By default the device is not protected */
           np->is_protected = false;
           INIT_LIST_HEAD(&np->domain_list);
+        INIT_LIST_HEAD(&np->deferred_probe);

I am not entirely happy to add a new list_head field per node just for the benefits of boot code. Could we re-use domain_list (with a comment in the code and appropriate ASSERT)?

Agree that only boot code uses deferred_probe field. I will consider re-using domain_list. Could you please clarify regarding ASSERT (where to put and what to check).

Quoted 10 lines
Quoted 10 lines
xen/common/device_tree.c +0 -0
             if ( new_format )
           {
xen/drivers/passthrough/arm/iommu.c +0 -0
index 2135233..3195919 100644
@@ -20,6 +20,12 @@
   #include <xen/device_tree.h>
   #include <asm/device.h>
Quoted 8 lines
Quoted 6 lines

  +/*
+ * Used to keep track of devices for which driver requested deferred probing
+ * (returns -EAGAIN).
+ */
+static LIST_HEAD(deferred_probe_list);

This wants to be in init section as this is only used at boot.

Will do.

Quoted 23 lines
Quoted 23 lines

+
  static const struct iommu_ops *iommu_ops;
    const struct iommu_ops *iommu_get_ops(void)

xen/drivers/passthrough/arm/iommu.c +1 -1
@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops)
     int __init iommu_hardware_setup(void)
   {
-    struct dt_device_node *np;
+    struct dt_device_node *np, *tmp;
       int rc;
       unsigned int num_iommus = 0;
   @@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void)
           rc = device_init(np, DEVICE_IOMMU, NULL);

          if ( !rc )
              num_iommus++;
+        else if (rc == -EAGAIN)
+            /*
+             * Driver requested deferred probing, so add this device to +             * the deferred list for further processing.
+             */
+            list_add(&np->deferred_probe, &deferred_probe_list); +    }

Quoted 8 lines
Quoted 6 lines

+
+    /*
+     * Process devices in the deferred list if at least one successfully
+     * probed device is present.
+     */

I think this can turn into an infinite loop if all device in deferred_probe_list still return -EDEFER_PROBE and num_iommus is a non-zero.

Agree.

Quoted 2 lines

A better condition would be to check that at least one IOMMU is added at each loop. If not, then we should bail with an error because it likely means something is buggy.

Sounds reasonable. Will do.

Just to clarify:

Quoted 1 line
Quoted 1 line
Quoted 1 line

A better condition would be to check that at least one IOMMU is added at each loop.

Maybe, not only added (rc == 0), but driver didn't request deferred probe (rc != -EAGAIN).

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#7 lore raw

Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request

Hi Oleksandr,

Quoted 43 lines · Oleksandr
Quoted 29 lines · Julien Grall
Quoted 28 lines · Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.

In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen -EAGAIN to be used for indicating that device probing is deferred.

This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.

---
  xen/common/device_tree.c            |  1 +
  xen/drivers/passthrough/arm/iommu.c | 35
++++++++++++++++++++++++++++++++++-
  xen/include/asm-arm/device.h        |  6 +++++-
  xen/include/xen/device_tree.h       |  1 +

  4 files changed, 41 insertions(+), 2 deletions(-)
Quoted 12 lines

xen/common/device_tree.c static unsigned long __init · line 1778

Quoted 10 lines
xen/common/device_tree.c +1 -0
index e107c6f..6f37448 100644
@@ -1774,6 +1774,7 @@ static unsigned long __init 
 unflatten_dt_node(const void *fdt,
           /* By default the device is not protected */
           np->is_protected = false;
           INIT_LIST_HEAD(&np->domain_list);
+        INIT_LIST_HEAD(&np->deferred_probe);

I am not entirely happy to add a new list_head field per node just for the benefits of boot code. Could we re-use domain_list (with a comment in the code and appropriate ASSERT)?

Quoted 1 line

Agree that only boot code uses deferred_probe field. I will consider re-using domain_list. Could you please clarify regarding ASSERT (where to put and what to check).

What I meant is adding an ASSERT to check that np->domain_list is at empty at least before trying to add in the list. This would help to debug any potential issue if we end up to use domain_list earlier in the future. I can't see why it would as iommu is called earlier, but who knows :).

Quoted 65 lines
Quoted 12 lines
Quoted 12 lines
xen/common/device_tree.c +0 -0
             if ( new_format )
           {
xen/drivers/passthrough/arm/iommu.c +0 -0
index 2135233..3195919 100644
@@ -20,6 +20,12 @@
   #include <xen/device_tree.h>
   #include <asm/device.h>
Quoted 8 lines
Quoted 6 lines

  +/*
+ * Used to keep track of devices for which driver requested deferred probing
+ * (returns -EAGAIN).
+ */
+static LIST_HEAD(deferred_probe_list);

This wants to be in init section as this is only used at boot.

Will do.

Quoted 23 lines
Quoted 23 lines

+
  static const struct iommu_ops *iommu_ops;
    const struct iommu_ops *iommu_get_ops(void)

xen/drivers/passthrough/arm/iommu.c +1 -1
@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops)
     int __init iommu_hardware_setup(void)
   {
-    struct dt_device_node *np;
+    struct dt_device_node *np, *tmp;
       int rc;
       unsigned int num_iommus = 0;
   @@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void)
           rc = device_init(np, DEVICE_IOMMU, NULL);

          if ( !rc )
              num_iommus++;
+        else if (rc == -EAGAIN)
+            /*
+             * Driver requested deferred probing, so add this device to +             * the deferred list for further processing.
+             */
+            list_add(&np->deferred_probe, &deferred_probe_list); +    }

Quoted 8 lines
Quoted 6 lines

+
+    /*
+     * Process devices in the deferred list if at least one successfully
+     * probed device is present.
+     */

I think this can turn into an infinite loop if all device in deferred_probe_list still return -EDEFER_PROBE and num_iommus is a non-zero.

Agree.

Quoted 2 lines

A better condition would be to check that at least one IOMMU is added at each loop. If not, then we should bail with an error because it likely means something is buggy.

Sounds reasonable. Will do.

Quoted 6 lines

Just to clarify:

Quoted 1 line
Quoted 1 line
Quoted 1 line

A better condition would be to check that at least one IOMMU is

added at each loop.

Maybe, not only added (rc == 0), but driver didn't request deferred probe (rc != -EAGAIN).

I think adding an IOMMU is enough. If you return an error other than -EAGAIN here after deferring probing, then you are likely going to fail at the next loop. So better to stop early.

I realize this not what the current code is doing (I know I wrote it ;)). But I am not sure it is sane to continue if only part of the IOMMUs are initialized. Most likely you will see an error much later that may be not trivial to find out.

Imagine you want to passthrough you network card to a guest but the IOMMU initialization failed...

Cheers,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#8 lore raw

Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request

Quoted 1 line · Julien Grall

Hi Oleksandr,

Hi, Julien

Quoted 45 lines · Oleksandr
Quoted 43 lines · Julien Grall
Quoted 41 lines · Oleksandr Tyshchenko
Quoted 33 lines

From: Oleksandr Tyshchenko <[email protected]>

This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.

In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen -EAGAIN to be used for indicating that device probing is deferred.

This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.

---
  xen/common/device_tree.c            |  1 +
  xen/drivers/passthrough/arm/iommu.c | 35
++++++++++++++++++++++++++++++++++-
  xen/include/asm-arm/device.h        |  6 +++++-
  xen/include/xen/device_tree.h       |  1 +

  4 files changed, 41 insertions(+), 2 deletions(-)
xen/common/device_tree.c +0 -0
index e107c6f..6f37448 100644

xen/common/device_tree.c static unsigned long __init · line 1778

Quoted 5 lines
xen/common/device_tree.c +1 -0
@@ -1774,6 +1774,7 @@ static unsigned long __init 
 unflatten_dt_node(const void *fdt,
           /* By default the device is not protected */
           np->is_protected = false;
           INIT_LIST_HEAD(&np->domain_list);
+        INIT_LIST_HEAD(&np->deferred_probe);

I am not entirely happy to add a new list_head field per node just for the benefits of boot code. Could we re-use domain_list (with a comment in the code and appropriate ASSERT)?

Quoted 3 lines
Quoted 1 line

Agree that only boot code uses deferred_probe field. I will consider re-using domain_list. Could you please clarify regarding ASSERT (where to put and what to check).

What I meant is adding an ASSERT to check that np->domain_list is at empty at least before trying to add in the list. This would help to debug any potential issue if we end up to use domain_list earlier in the future. I can't see why it would as iommu is called earlier, but who knows :).

Got it. Thank you for clarification.

Quoted 43 lines
Quoted 43 lines
Quoted 23 lines
Quoted 23 lines
xen/common/device_tree.c +1 -0
+
   static const struct iommu_ops *iommu_ops;

    const struct iommu_ops *iommu_get_ops(void)

xen/common/device_tree.c +1 -1
@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops 
 *ops)
     int __init iommu_hardware_setup(void)
   {
-    struct dt_device_node *np;
+    struct dt_device_node *np, *tmp;
       int rc;
       unsigned int num_iommus = 0;
   @@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void)

          rc = device_init(np, DEVICE_IOMMU, NULL);
          if ( !rc )
              num_iommus++;
+        else if (rc == -EAGAIN)
+            /*
+             * Driver requested deferred probing, so add this device to
+             * the deferred list for further processing. +             */
+            list_add(&np->deferred_probe, &deferred_probe_list); +    }

Quoted 8 lines
Quoted 6 lines

+
+    /*
+     * Process devices in the deferred list if at least one successfully
+     * probed device is present.
+     */

I think this can turn into an infinite loop if all device in deferred_probe_list still return -EDEFER_PROBE and num_iommus is a non-zero.

Agree.

Quoted 2 lines

A better condition would be to check that at least one IOMMU is added at each loop. If not, then we should bail with an error because it likely means something is buggy.

Sounds reasonable. Will do.

Just to clarify:

Quoted 7 lines
Quoted 3 lines

 >>> A better condition would be to check that at least one IOMMU is added at each loop.

Maybe, not only added (rc == 0), but driver didn't request deferred probe (rc != -EAGAIN).

I think adding an IOMMU is enough. If you return an error other than -EAGAIN here after deferring probing, then you are likely going to fail at the next loop. So better to stop early.

It makes sense.

Quoted 5 lines

I realize this not what the current code is doing (I know I wrote it ;)). But I am not sure it is sane to continue if only part of the IOMMUs are initialized. Most likely you will see an error much later that may be not trivial to find out.

Imagine you want to passthrough you network card to a guest but the IOMMU initialization failed...

Oh, agree.

As I understand, the new strict logic would be the following:

If initialization for at least one IOMMU device failed (device_init returns an error other than -EAGAIN), we should stop and return an error to upper layer (even if num_iommus > 0). No matter whether it is during the first attempt or after deferring probe. We don't allow the "I/O virtualisation" to be enabled (iommu_enabled == true) with only part of the IOMMU devices being initialized. Is my understanding correct?

Quoted 5 lines

Cheers,

Signature · 1 line
Julien Grall
Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#9 lore raw

Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request

Hi Oleksandr,

Quoted 111 lines · Oleksandr
Quoted 1 line · Julien Grall

Hi Oleksandr,

Hi, Julien

Quoted 44 lines · Oleksandr
Quoted 42 lines · Julien Grall
Quoted 40 lines · Oleksandr Tyshchenko
Quoted 32 lines

From: Oleksandr Tyshchenko <[email protected]>

This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.

In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen -EAGAIN to be used for indicating that device probing is deferred.

This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.

---
  xen/common/device_tree.c            |  1 + xen/drivers/passthrough/arm/iommu.c | 35 ++++++++++++++++++++++++++++++++++- xen/include/asm-arm/device.h        |  6 +++++- xen/include/xen/device_tree.h       |  1 +

  4 files changed, 41 insertions(+), 2 deletions(-)
xen/common/device_tree.c +0 -0
index e107c6f..6f37448 100644

xen/common/device_tree.c static unsigned long __init unflatten_dt_node(const · line 1778

Quoted 5 lines
xen/common/device_tree.c +1 -0
@@ -1774,6 +1774,7 @@ static unsigned long __init unflatten_dt_node(const 
 void *fdt,
           /* By default the device is not protected */
           np->is_protected = false;
           INIT_LIST_HEAD(&np->domain_list);
+        INIT_LIST_HEAD(&np->deferred_probe);

I am not entirely happy to add a new list_head field per node just for the benefits of boot code. Could we re-use domain_list (with a comment in the code and appropriate ASSERT)?

Quoted 3 lines
Quoted 1 line

Agree that only boot code uses deferred_probe field. I will consider re-using domain_list. Could you please clarify regarding ASSERT (where to put and what to check).

What I meant is adding an ASSERT to check that np->domain_list is at empty at least before trying to add in the list. This would help to debug any potential issue if we end up to use domain_list earlier in the future. I can't see why it would as iommu is called earlier, but who knows :).

Got it. Thank you for clarification.

Quoted 38 lines
Quoted 38 lines
Quoted 20 lines
Quoted 20 lines
xen/common/device_tree.c +1 -0
+
   static const struct iommu_ops *iommu_ops;

    const struct iommu_ops *iommu_get_ops(void)

xen/common/device_tree.c +1 -1
@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops)
     int __init iommu_hardware_setup(void)
   {
-    struct dt_device_node *np;
+    struct dt_device_node *np, *tmp;
       int rc;
       unsigned int num_iommus = 0;
   @@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void)
           rc = device_init(np, DEVICE_IOMMU, NULL);

          if ( !rc )
              num_iommus++;
+        else if (rc == -EAGAIN)
+            /*
+             * Driver requested deferred probing, so add this device to +             * the deferred list for further processing.
+             */
+            list_add(&np->deferred_probe, &deferred_probe_list);

Quoted 8 lines
Quoted 6 lines

+    }
+
+    /*
+     * Process devices in the deferred list if at least one successfully +     * probed device is present.
+     */

I think this can turn into an infinite loop if all device in deferred_probe_list still return -EDEFER_PROBE and num_iommus is a non-zero.

Agree.

Quoted 2 lines

A better condition would be to check that at least one IOMMU is added at each loop. If not, then we should bail with an error because it likely means something is buggy.

Sounds reasonable. Will do.

Quoted 7 lines
Quoted 5 lines

Just to clarify:

 >>> A better condition would be to check that at least one IOMMU is added at each loop.

Maybe, not only added (rc == 0), but driver didn't request deferred probe (rc != -EAGAIN).

I think adding an IOMMU is enough. If you return an error other than -EAGAIN here after deferring probing, then you are likely going to fail at the next loop. So better to stop early.

It makes sense.

Quoted 4 lines

I realize this not what the current code is doing (I know I wrote it ;)). But I am not sure it is sane to continue if only part of the IOMMUs are initialized. Most likely you will see an error much later that may be not trivial to find out.

Quoted 7 lines
Quoted 1 line

Imagine you want to passthrough you network card to a guest but the IOMMU initialization failed...

Oh, agree.

As I understand, the new strict logic would be the following:

If initialization for at least one IOMMU device failed (device_init returns an error other than -EAGAIN), we should stop and return an error to upper layer (even if num_iommus > 0). No matter whether it is during the first attempt or after deferring probe. We don't allow the "I/O virtualisation" to be enabled (iommu_enabled == true) with only part of the IOMMU devices being initialized. Is my understanding correct?

Let me summarize the discussion we had on IRC :). Without your patch, Xen may initialize only half the IOMMUs. If the device is behind an IOMMU that wasn't initialized, then we have two possibility:
1) The device was already mark as protected (if using the old binding in the
SMMU). Xen will not be able to assign the device to Dom0 and therefore Xen will crash (not able to build dom0). For domU, it will depend whether the configuration contain the options 'dtdev'. If the option is specified, then guest will fail to build. On the contrary if the option isn't specified then the guest will boot and this could either lead to transaction failure (if the IOMMU was already reset) or bypassing the IOMMU. Note that the latter can today happen if your IOMMU was disabled. But we can't do much against it.
2) The device is not marked as protected. Xen will not be able to "assign"
the device to Dom0 and this could either lead to the device bypassing the IOMMU or a transaction failure. For domU, the problem is similar to 1).

In the case of the SMMU driver, we only support old bindings. So devices are marked as protected during SMMU initialization. This is done before the SMMU is reset. Before reset the SMMU will bypassed.

So the risk is to have an half secure system and may be unnoticed until later. I realize this is the current behavior, so not very ideal.

It feels to me if the user requested to use IOMMU then if we should panic if any of the available IOMMU are not initialized correctly. This will save a lot of debug afterwards.

@Stefano, any opinions?

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#10 lore raw

Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request

Quoted 169 lines · Julien Grall

Hi Oleksandr,

Quoted 138 lines · Oleksandr
Quoted 1 line · Julien Grall

Hi Oleksandr,

Hi, Julien

Quoted 47 lines · Oleksandr
Quoted 45 lines · Julien Grall
Quoted 43 lines · Oleksandr Tyshchenko
Quoted 33 lines

From: Oleksandr Tyshchenko <[email protected]>

This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.

In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen -EAGAIN to be used for indicating that device probing is deferred.

This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.

---
  xen/common/device_tree.c            |  1 +
  xen/drivers/passthrough/arm/iommu.c | 35
++++++++++++++++++++++++++++++++++-
  xen/include/asm-arm/device.h        |  6 +++++-
  xen/include/xen/device_tree.h       |  1 +

  4 files changed, 41 insertions(+), 2 deletions(-)
xen/common/device_tree.c +0 -0
index e107c6f..6f37448 100644

xen/common/device_tree.c static unsigned long __init · line 1778

Quoted 5 lines
xen/common/device_tree.c +1 -0
@@ -1774,6 +1774,7 @@ static unsigned long __init
 unflatten_dt_node(const void *fdt,
           /* By default the device is not protected */
           np->is_protected = false;
           INIT_LIST_HEAD(&np->domain_list);
+        INIT_LIST_HEAD(&np->deferred_probe);

I am not entirely happy to add a new list_head field per node just for the benefits of boot code. Could we re-use domain_list (with a comment in the code and appropriate ASSERT)?

Quoted 8 lines
Quoted 3 lines

Agree that only boot code uses deferred_probe field. I will consider re-using domain_list. Could you please clarify regarding ASSERT (where to put and what to check).

What I meant is adding an ASSERT to check that np->domain_list is at empty at least before trying to add in the list. This would help to debug any potential issue if we end up to use domain_list earlier in the future. I can't see why it would as iommu is called earlier, but who knows :).

Got it. Thank you for clarification.

Quoted 50 lines
Quoted 50 lines
Quoted 25 lines
Quoted 25 lines
xen/common/device_tree.c +1 -0
+
   static const struct iommu_ops *iommu_ops;

    const struct iommu_ops *iommu_get_ops(void)

xen/common/device_tree.c +1 -1
@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops
 *ops)
     int __init iommu_hardware_setup(void)
   {
-    struct dt_device_node *np;
+    struct dt_device_node *np, *tmp;
       int rc;
       unsigned int num_iommus = 0;
   @@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void)

          rc = device_init(np, DEVICE_IOMMU, NULL);
          if ( !rc )
              num_iommus++;
+        else if (rc == -EAGAIN)
+            /*
+             * Driver requested deferred probing, so add this device to
+             * the deferred list for further processing. +             */
+            list_add(&np->deferred_probe, &deferred_probe_list); +    }
+
+    /*

Quoted 8 lines
Quoted 4 lines

+     * Process devices in the deferred list if at least one successfully
+     * probed device is present.
+     */

I think this can turn into an infinite loop if all device in deferred_probe_list still return -EDEFER_PROBE and num_iommus is a non-zero.

Agree.

Quoted 4 lines

A better condition would be to check that at least one IOMMU is added at each loop. If not, then we should bail with an error because it likely means something is buggy.

Sounds reasonable. Will do.

Just to clarify:

 >>> A better condition would be to check that at least one IOMMU is added at each loop.

Quoted 6 lines
Quoted 2 lines

Maybe, not only added (rc == 0), but driver didn't request deferred probe (rc != -EAGAIN).

I think adding an IOMMU is enough. If you return an error other than -EAGAIN here after deferring probing, then you are likely going to fail at the next loop. So better to stop early.

It makes sense.

Quoted 9 lines

I realize this not what the current code is doing (I know I wrote it ;)). But I am not sure it is sane to continue if only part of the IOMMUs are initialized. Most likely you will see an error much later that may be not trivial to find out.

Imagine you want to passthrough you network card to a guest but the IOMMU initialization failed...

Oh, agree.

Quoted 8 lines

As I understand, the new strict logic would be the following:

If initialization for at least one IOMMU device failed (device_init returns an error other than -EAGAIN), we should stop and return an error to upper layer (even if num_iommus > 0). No matter whether it is during the first attempt or after deferring probe. We don't allow the "I/O virtualisation" to be enabled (iommu_enabled == true) with only part of the IOMMU devices being initialized. Is my understanding correct?

Let me summarize the discussion we had on IRC :). Without your patch, Xen may initialize only half the IOMMUs. If the device is behind an IOMMU that wasn't initialized, then we have two possibility:
1) The device was already mark as protected (if using the old binding in
the SMMU). Xen will not be able to assign the device to Dom0 and therefore Xen will crash (not able to build dom0). For domU, it will depend whether the configuration contain the options 'dtdev'. If the option is specified, then guest will fail to build. On the contrary if the option isn't specified then the guest will boot and this could either lead to transaction failure (if the IOMMU was already reset) or bypassing the IOMMU. Note that the latter can today happen if your IOMMU was disabled. But we can't do much against it.
2) The device is not marked as protected. Xen will not be able to "assign"
the device to Dom0 and this could either lead to the device bypassing the IOMMU or a transaction failure. For domU, the problem is similar to 1).

In the case of the SMMU driver, we only support old bindings. So devices are marked as protected during SMMU initialization. This is done before the SMMU is reset. Before reset the SMMU will bypassed.

Quoted 8 lines

So the risk is to have an half secure system and may be unnoticed until later. I realize this is the current behavior, so not very ideal.

It feels to me if the user requested to use IOMMU then if we should panic if any of the available IOMMU are not initialized correctly. This will save a lot of debug afterwards.

@Stefano, any opinions?

I agree that we should enable all IOMMUs or none. I don't think we want to deal with partially initialized IOMMUs systems.

Failure to enable all IOMMUs should lead to returning an error from the relevant function (arch_iommu_populate_page_table?) which should translate into Xen failing to boot and crashing. Which I think it is what you are suggesting, right?

(I wouldn't call panic() inside the IOMMU specific initializer, because there might be iommu= command line options for Xen that specify a different desired outcome. I would deal with this case the same way we would deal with an error during initialization of a single IOMMU.)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#11 lore raw

Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request

Hi Stefano,

Quoted 6 lines · Stefano Stabellini

On Wed, 14 Aug 2019, Julien Grall wrote:
I agree that we should enable all IOMMUs or none. I don't think we want to deal with partially initialized IOMMUs systems.

Failure to enable all IOMMUs should lead to returning an error from the relevant function (arch_iommu_populate_page_table?) which should

The patch is:

|> start_xen()
|>   iommu_setup()
|>     iommu_hardware_setup()
Quoted 2 lines

translate into Xen failing to boot and crashing. Which I think it is what you are suggesting, right?

That's correct. At the moment the return value of iommu_setup() is ignored. What I would like to suggest is something along the following lines:

rc = iommu_setup();
if ( iommu_enable && rc != -ENODEV )
panic("Unable to setup IOMMUs");

Quoted 5 lines

(I wouldn't call panic() inside the IOMMU specific initializer, because there might be iommu= command line options for Xen that specify a different desired outcome. I would deal with this case the same way we would deal with an error during initialization of a single IOMMU.)

I am not sure to understand this. If you have an half initialized IOMMU (note the "single" here!), then continuing is likely to make things much worst.

I don't advocate to put the panic() inside the IOMMU specific initializer (see above). But clearly, we should return an error no matter the content of 'iommu' command line if the user requested to enable the IOMMUs (if any). It wouldn't be right if the user can say "ignore IOMMU error" as most likely you will have unexpected error afterwards.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#12 lore raw

Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request

Hi,

Quoted 30 lines · Julien Grall
Quoted 6 lines · Stefano Stabellini

On Wed, 14 Aug 2019, Julien Grall wrote:
I agree that we should enable all IOMMUs or none. I don't think we want to deal with partially initialized IOMMUs systems.

Failure to enable all IOMMUs should lead to returning an error from the relevant function (arch_iommu_populate_page_table?) which should

The patch is:

|> start_xen()
|>   iommu_setup()
|>     iommu_hardware_setup()
Quoted 2 lines

translate into Xen failing to boot and crashing. Which I think it is what you are suggesting, right?

That's correct. At the moment the return value of iommu_setup() is ignored. What I would like to suggest is something along the following lines:

rc = iommu_setup();
if ( iommu_enable && rc != -ENODEV )
  panic("Unable to setup IOMMUs");

Quoted 5 lines

(I wouldn't call panic() inside the IOMMU specific initializer, because there might be iommu= command line options for Xen that specify a different desired outcome. I would deal with this case the same way we would deal with an error during initialization of a single IOMMU.)

Quoted 8 lines

I am not sure to understand this. If you have an half initialized IOMMU (note the "single" here!), then continuing is likely to make things much worst.

I don't advocate to put the panic() inside the IOMMU specific initializer (see above). But clearly, we should return an error no matter the content of 'iommu' command line if the user requested to enable the IOMMUs (if any). It wouldn't be right if the user can say "ignore IOMMU error" as most likely you will have unexpected error afterwards.

I noticed there was already a panic() in iommu_setup() just in case the user force the use of IOMMU but they were not initialized. I was half-tempted to set iommu_force to true for Arm, but I think this is a different issue.

So here my take (not tested nor compiled).

xen/arch/arm/setup.c +4 -1
index 2c5d1372c0..8f94f618b0 100644
@@ -755,6 +755,7 @@ void __init start_xen(unsigned long boot_phys_offset,
         .max_grant_frames = gnttab_dom0_frames(),
         .max_maptrack_frames = opt_max_maptrack_frames,
     };
+    int rc;
 
     dcache_line_bytes = read_dcache_line_bytes();
 
@@ -890,7 +891,9 @@ void __init start_xen(unsigned long boot_phys_offset,
 
     setup_virt_paging();
 
-    iommu_setup();
+    rc = iommu_setup();
+    if ( !iommu_enabled && rc != -ENODEV )
+        panic("Couldn't configure correctly all the IOMMUs.");
 
     do_initcalls();
 
xen/drivers/passthrough/arm/iommu.c +8 -0
index 2135233736..f219de9ac3 100644
@@ -51,6 +51,14 @@ int __init iommu_hardware_setup(void)
         rc = device_init(np, DEVICE_IOMMU, NULL);
         if ( !rc )
             num_iommus++;
+        /*
+         * Ignore the following error codes:
+         *   - EBADF: Indicate the current not is not an IOMMU
+         *   - ENODEV: The IOMMU is not present or cannot be used by
+         *     Xen.
+         */
+        else if ( rc != -EBADF && rc != -ENODEV )
+            return rc;
     }
 
     return ( num_iommus > 0 ) ? 0 : -ENODEV;
Quoted 3 lines

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#13 lore raw

Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request

Quoted 1 line · Julien Grall

Hi,

Hi Julien

Quoted 5 lines

I noticed there was already a panic() in iommu_setup() just in case the user force the use of IOMMU but they were not initialized. I was half-tempted to set iommu_force to true for Arm, but I think this is a different issue.

So here my take (not tested nor compiled).

Thank you. I will check it and come back with results.

Quoted 49 lines
xen/arch/arm/setup.c +4 -1
index 2c5d1372c0..8f94f618b0 100644
@@ -755,6 +755,7 @@ void __init start_xen(unsigned long boot_phys_offset,
          .max_grant_frames = gnttab_dom0_frames(),
          .max_maptrack_frames = opt_max_maptrack_frames,
      };
+    int rc;
 
      dcache_line_bytes = read_dcache_line_bytes();
 
@@ -890,7 +891,9 @@ void __init start_xen(unsigned long boot_phys_offset,
 
      setup_virt_paging();
 
-    iommu_setup();
+    rc = iommu_setup();
+    if ( !iommu_enabled && rc != -ENODEV )
+        panic("Couldn't configure correctly all the IOMMUs.");
 
      do_initcalls();
 
xen/drivers/passthrough/arm/iommu.c +8 -0
index 2135233736..f219de9ac3 100644
@@ -51,6 +51,14 @@ int __init iommu_hardware_setup(void)
          rc = device_init(np, DEVICE_IOMMU, NULL);
          if ( !rc )
              num_iommus++;
+        /*
+         * Ignore the following error codes:
+         *   - EBADF: Indicate the current not is not an IOMMU
+         *   - ENODEV: The IOMMU is not present or cannot be used by
+         *     Xen.
+         */
+        else if ( rc != -EBADF && rc != -ENODEV )
+            return rc;
      }
 
      return ( num_iommus > 0 ) ? 0 : -ENODEV;
Quoted 6 lines

Cheers,

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#14 lore raw

Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request

Hi, Julien

Quoted 8 lines
Quoted 6 lines

I noticed there was already a panic() in iommu_setup() just in case the user force the use of IOMMU but they were not initialized. I was half-tempted to set iommu_force to true for Arm, but I think this is a different issue.

So here my take (not tested nor compiled).

Thank you. I will check it and come back with results.

I have preliminary tested it with my IPMMU series including new modification of "deferred probing" patch (attached). Being honest my series is not based on the *current* staging, but not too outdated.

So, your patch works as expected in the following scenarios:

1. [No panic] Without IOMMU driver in Xen (#CONFIG_IPMMU_VMSA is not set).

2. [No panic] IOMMU driver is present, but reports it is cannot be used in Xen (P2M sharing not supported, etc) by returning -ENODEV.

3. [No panic] IOMMU is globally disabled in command line "iommu=0".

4. [No panic] IOMMU driver requests deferred probing until the last device in DT is initialized, after that all deferred devices get initialized.

5. [Panic] IOMMU driver returns an error (other than -EBADF and -ENODEV) at the very beginning/when a part of devices are already initialized.

6. [Panic] IOMMU driver always returns deferred probing/returns an error other than -EAGAIN after deferred probing.

Quoted 10 lines
Quoted 10 lines
xen/arch/arm/setup.c +0 -0
index 2c5d1372c0..8f94f618b0 100644
@@ -755,6 +755,7 @@ void __init start_xen(unsigned long 
 boot_phys_offset,
           .max_grant_frames = gnttab_dom0_frames(),
           .max_maptrack_frames = opt_max_maptrack_frames,
       };
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/setup.c +1 -0
+    int rc;
         dcache_line_bytes = read_dcache_line_bytes();
   @@ -890,7 +891,9 @@ void __init start_xen(unsigned long boot_phys_offset,

        setup_virt_paging();
  -    iommu_setup();
+    rc = iommu_setup();
+    if ( !iommu_enabled && rc != -ENODEV )
+        panic("Couldn't configure correctly all the IOMMUs.");

"\n" should be added.

Signature · 3 lines
Regards,

Oleksandr Tyshchenko

From ff0af4d3d332a4b7b1b9b7edd4dbd1b3a263cb70 Mon Sep 17 00:00:00 2001
From: Oleksandr Tyshchenko <[email protected]>
Date: Thu, 15 Aug 2019 16:56:04 +0300
Subject: [PATCH] iommu/arm: Add ability to handle deferred probing request

This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.

In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen -EAGAIN to be used for indicating that device probing is deferred.

This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.

While here, order the headers alphabetically.

---
Changes V2 -> V3:
- removed deferred_probe field from struct dt_device_node, re-used domain_list instead
- documented domain_list usage
- added ASSERT to check that np->domain_list is empty before re-using it
- put deferred_probe_list to init section
- used more strict logic regarding processing devices in the deferred list
- added more comments to code
- put headers in alphabetical order

---
 xen/drivers/passthrough/arm/iommu.c | 55 ++++++++++++++++++++++++++++++++++---
 xen/include/asm-arm/device.h        |  6 +++-
 xen/include/xen/device_tree.h       |  7 +++++
 3 files changed, 63 insertions(+), 5 deletions(-)
xen/drivers/passthrough/arm/iommu.c +51 -4
index f8bb2c3..8569d2e 100644
@@ -15,11 +15,20 @@
  * GNU General Public License for more details.
  */
 
-#include <xen/lib.h>
-#include <xen/iommu.h>
 #include <xen/device_tree.h>
+#include <xen/iommu.h>
+#include <xen/lib.h>
+
 #include <asm/device.h>
 
+/*
+ * Deferred probe list is used to keep track of devices for which driver
+ * requested deferred probing (returned -EAGAIN).
+ *
+ * We re-use device's domain_list to link the device in the deferred list.
+ */
+static __initdata LIST_HEAD(deferred_probe_list);
+
 static const struct iommu_ops *iommu_ops;
 
 const struct iommu_ops *iommu_get_ops(void)
@@ -42,7 +51,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops)
 
 int __init iommu_hardware_setup(void)
 {
-    struct dt_device_node *np;
+    struct dt_device_node *np, *tmp;
     int rc;
     unsigned int num_iommus = 0;
 
@@ -51,6 +60,17 @@ int __init iommu_hardware_setup(void)
         rc = device_init(np, DEVICE_IOMMU, NULL);
         if ( !rc )
             num_iommus++;
+        else if ( rc == -EAGAIN )
+        {
+            /* We expect nobody uses domain_list at such early stage. */
+            ASSERT(list_empty(&np->domain_list));
+
+            /*
+             * Driver requested deferred probing, so add this device to
+             * the deferred list for further processing.
+             */
+            list_add(&np->domain_list, &deferred_probe_list);
+        }
         /*
          * Ignore the following error codes:
          *   - EBADF: Indicate the current is not an IOMMU
@@ -60,7 +80,34 @@ int __init iommu_hardware_setup(void)
             return rc;
     }
 
-    return ( num_iommus > 0 ) ? 0 : -ENODEV;
+    rc = ( num_iommus > 0 ) ? 0 : -ENODEV;
+
+    /*
+     * Process devices in the deferred list if it is not empty and at least one
+     * successfully probed device is present. Check that at least one device
+     * is added at each loop, otherwise we may get an infinite loop. Also stop
+     * processing if we got an error other than -EAGAIN.
+     */
+    while ( !list_empty(&deferred_probe_list) && num_iommus )
+    {
+        num_iommus = 0;
+
+        list_for_each_entry_safe ( np, tmp, &deferred_probe_list, domain_list )
+        {
+            rc = device_init(np, DEVICE_IOMMU, NULL);
+            if ( !rc )
+            {
+                num_iommus++;
+
+                /* Remove successfully probed device from the deferred list. */
+                list_del_init(&np->domain_list);
+            }
+            else if ( rc != -EAGAIN )
+                return rc;
+        }
+    }
+
+    return rc;
 }
 
 void __hwdom_init arch_iommu_check_autotranslated_hwdom(struct domain *d)
xen/include/asm-arm/device.h +5 -1
index 63a0f36..ee1c3bc 100644
@@ -44,7 +44,11 @@ struct device_desc {
     enum device_class class;
     /* List of devices supported by this driver */
     const struct dt_device_match *dt_match;
-    /* Device initialization */
+    /*
+     * Device initialization.
+     *
+     * -EAGAIN is used to indicate that device probing is deferred.
+     */
     int (*init)(struct dt_device_node *dev, const void *data);
 };
 
xen/include/xen/device_tree.h +7 -0
index e1ec6cb..60ae99c 100644
@@ -92,6 +92,13 @@ struct dt_device_node {
 
     /* IOMMU specific fields */
     bool is_protected;
+    /*
+     * The main purpose of this list node is to link the structure in the list
+     * of devices assigned to domain.
+     *
+     * Boot code (iommu_hardware_setup) re-uses this list to link the structure
+     * in the list of devices for which driver requested deferred probing.
+     */
     struct list_head domain_list;
 
     struct device dev;
Signature · 1 line
2.7.4

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#15 lore raw

RFC PATCH v2 3/6 xen/common: Introduce _xrealloc function

In reply to Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

Next patch in this series will make use of it.

Original patch was initially posted by Sameer Goel:
https://lists.xen.org/archives/html/xen-devel/2017-06/msg00858.html

This could be considered as another attempt to add it:
https://www.mail-archive.com/[email protected]/msg21335.html

CC Andrew Cooper <[email protected]>
CC George Dunlap <[email protected]>
CC Ian Jackson <[email protected]>
CC Jan Beulich <[email protected]>
CC Julien Grall <[email protected]>
CC Konrad Rzeszutek Wilk <[email protected]>
CC Stefano Stabellini <[email protected]>
CC Tim Deegan <[email protected]>
CC Wei Liu <[email protected]>

---
[As it was previously discussed with Julien in IRC]

The reason for this patch to be an RFC is that patch itself is not completely correct and I don't fully understand what/how should be done for this patch to be accepted. Or whether community even wants this to go in. So, to avoid bike shedding, the first target is to collect feedback from the maintainers.

In a nutshell, the upcoming "iommu_fwspec" support on ARM is going to use xrealloc when adding new device ID.

We really want to have "iommu_fwspec" support which will give us a generic abstract way to add new device to the IOMMU based on the generic IOMMU DT binding.

This is how Linux does:
https://github.com/torvalds/linux/blob/master/drivers/iommu/iommu.c#L2257
and we are doing the similar in next patch of this thread:
"iommu/arm: Add lightweight iommu_fwspec support"

---
 xen/common/xmalloc_tlsf.c | 21 +++++++++++++++++++++
 xen/include/xen/xmalloc.h |  1 +
 2 files changed, 22 insertions(+)
xen/common/xmalloc_tlsf.c +21 -0
index 2076953..c080763 100644
@@ -610,6 +610,27 @@ void *_xzalloc(unsigned long size, unsigned long align)
     return p ? memset(p, 0, size) : p;
 }
 
+void *_xrealloc(void *p, unsigned long new_size, unsigned long align)
+{
+    void *new_p;
+
+    if ( !new_size )
+    {
+        xfree(p);
+        return NULL;
+    }
+
+    new_p = _xmalloc(new_size, align);
+
+    if ( new_p && p )
+    {
+        memcpy(new_p, p, new_size);
+        xfree(p);
+    }
+
+    return new_p;
+}
+
 void xfree(void *p)
 {
     struct bhdr *b;
xen/include/xen/xmalloc.h +1 -0
index b486fe4..63961ef 100644
@@ -51,6 +51,7 @@ extern void xfree(void *);
 /* Underlying functions */
 extern void *_xmalloc(unsigned long size, unsigned long align);
 extern void *_xzalloc(unsigned long size, unsigned long align);
+extern void *_xrealloc(void *p, unsigned long new_size, unsigned long align);
 
 static inline void *_xmalloc_array(
     unsigned long size, unsigned long align, unsigned long num)
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#16 lore raw

Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function

Quoted 16 lines · Oleksandr Tyshchenko
xen/common/xmalloc_tlsf.c +10 -0
@@ -610,6 +610,27 @@ void *_xzalloc(unsigned long size, unsigned long align)
      return p ? memset(p, 0, size) : p;
  }
 
+void *_xrealloc(void *p, unsigned long new_size, unsigned long align)
+{
+    void *new_p;
+
+    if ( !new_size )
+    {
+        xfree(p);
+        return NULL;
+    }
+

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 632

Quoted 10 lines
xen/common/xmalloc_tlsf.c +10 -0
+    new_p = _xmalloc(new_size, align);
+
+    if ( new_p && p )
+    {
+        memcpy(new_p, p, new_size);
+        xfree(p);
+    }
+
+    return new_p;
+}

While I can see why having a re-allocation function may be handy, explicit / direct use of _xmalloc() and _xzalloc() are discouraged, in favor of the more type-safe underscore-less variants. I can't see though how a type-safe "realloc" could look like, except for arrays. If resizing arrays is all you're after, I'd like to recommend to go that route rather then the suggested one here. If resizing arbitrary objects is the goal, then what you suggest may be the only route, but I'd still be not overly happy to see such added.

Furthermore you don't even use internals of the allocator: It is common practice to avoid re-allocation if the requested size fits within the already allocated block. That's not the least helpful because in such a case you can't possibly suffer any -ENOMEM condition.

And finally - please note _xmalloc()'s and _xfree()'s use / special casing of ZERO_BLOCK_PTR: You absolutely would need to mirror this here.

Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#17 lore raw

Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function

On 05.08.19 13:02, Jan Beulich wrote:

Hi, Jan

Quoted 3 lines

While I can see why having a re-allocation function may be handy, explicit / direct use of _xmalloc() and _xzalloc() are discouraged, in favor of the more type-safe underscore-less variants.

took into account

Quoted 7 lines

I can't
see though how a type-safe "realloc" could look like, except for arrays. If resizing arrays is all you're after, I'd like to recommend to go that route rather then the suggested one here. If resizing arbitrary objects is the goal, then what you suggest may be the only route, but I'd still be not overly happy to see such added.

My main goal is to get "ported" from Linux "iommu_fwspec" support (xrealloc user) in [1].

I tried to retain code as much as possible while porting. So, this patch adds almost the same thing what the ported code expects.

But, I would be OK to consider modifying a code in a way to resize an array as well as any other variants if present.

Quoted 6 lines

Furthermore you don't even use internals of the allocator: It is common practice to avoid re-allocation if the requested size fits within the already allocated block. That's not the least helpful because in such a case you can't possibly suffer any -ENOMEM condition.

agree, took into account as well.

Quoted 4 lines

And finally - please note _xmalloc()'s and _xfree()'s use / special casing of ZERO_BLOCK_PTR: You absolutely would need to mirror this here.

got it, will use for zero-size allocation

[1] https://lists.xenproject.org/archives/html/xen-devel/2019-08/msg00257.html

Thank you.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#18 lore raw

Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function

Quoted 9 lines · Oleksandr
Quoted 7 lines · Jan Beulich

I can't
see though how a type-safe "realloc" could look like, except for arrays. If resizing arrays is all you're after, I'd like to recommend to go that route rather then the suggested one here. If resizing arbitrary objects is the goal, then what you suggest may be the only route, but I'd still be not overly happy to see such added.

Quoted 5 lines

My main goal is to get "ported" from Linux "iommu_fwspec" support (xrealloc user) in [1].

I tried to retain code as much as possible while porting. So, this patch adds almost the same thing what the ported code expects.

But, I would be OK to consider modifying a code in a way to resize an array as well as any other variants if present.

I've looked at the use in patch 4, and it really isn't an array allocation. Even a basic allocation would use _xmalloc() in this case (you'll find examples in the tree if you want). Nevertheless I'd appreciate if the type-unsafe _xrealloc() didn't remain the only re-allocation construct, as to avoiding people using it just because there's no better alternative.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#19 lore raw

Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function

Hi,

Quoted 4 lines

Nevertheless
I'd appreciate if the type-unsafe _xrealloc() didn't remain the only re-allocation construct, as to avoiding people using it just because there's no better alternative.

I got your point.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#20 lore raw

Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function

In reply to Jan Beulich

Hi Jan,

Quoted 29 lines · Jan Beulich
Quoted 16 lines · Oleksandr Tyshchenko
xen/common/xmalloc_tlsf.c +10 -0
@@ -610,6 +610,27 @@ void *_xzalloc(unsigned long size, unsigned long align)
      return p ? memset(p, 0, size) : p;
  }
 
+void *_xrealloc(void *p, unsigned long new_size, unsigned long align)
+{
+    void *new_p;
+
+    if ( !new_size )
+    {
+        xfree(p);
+        return NULL;
+    }
+

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 632

Quoted 10 lines
xen/common/xmalloc_tlsf.c +10 -0
+    new_p = _xmalloc(new_size, align);
+
+    if ( new_p && p )
+    {
+        memcpy(new_p, p, new_size);
+        xfree(p);
+    }
+
+    return new_p;
+}

While I can see why having a re-allocation function may be handy,

Quoted 8 lines

explicit / direct use of _xmalloc() and _xzalloc() are discouraged, in favor of the more type-safe underscore-less variants. I can't see though how a type-safe "realloc" could look like, except for arrays. If resizing arrays is all you're after, I'd like to recommend to go that route rather then the suggested one here. If resizing arbitrary objects is the goal, then what you suggest may be the only route, but I'd still be not overly happy to see such added.

I can see 3 uses for realloc:

a. re-allocate generic data buffer
b. re-allocate array
c. re-allocate struct with flexible buffer.

option c. is about structures like this:

struct arrlen

{
        size_t len;
        int data[1];
};

This is Oleksandr's case.

So for option a. we can use _xreallocate(ptr, size, align) For option b. we can use xrealloc_array(_ptr, _type, _num) And for option c. I propose to implement the following macro:

#define realloc_flex_struct(_ptr, _type, _field, _len) \
((_type *)_xrealloc(_ptr, offsetof(_type, _field[_len]) , __alignof__(_type)))

It can be used in the following way:

newptr = realloc_flex_struct(ptr, struct arrlen, newsize);

As you can see, this approach is type-safe and covers Oleksanrd's case.

Signature · 5 lines
Volodymyr Babchuk at EPAM
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#21 lore raw

Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function

Quoted 63 lines · Volodymyr Babchuk

Hi Jan,

Quoted 29 lines · Jan Beulich
Quoted 16 lines · Oleksandr Tyshchenko
xen/common/xmalloc_tlsf.c +10 -0
@@ -610,6 +610,27 @@ void *_xzalloc(unsigned long size, unsigned long align)
       return p ? memset(p, 0, size) : p;
   }
 
+void *_xrealloc(void *p, unsigned long new_size, unsigned long align)
+{
+    void *new_p;
+
+    if ( !new_size )
+    {
+        xfree(p);
+        return NULL;
+    }
+

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 632

Quoted 10 lines
xen/common/xmalloc_tlsf.c +10 -0
+    new_p = _xmalloc(new_size, align);
+
+    if ( new_p && p )
+    {
+        memcpy(new_p, p, new_size);
+        xfree(p);
+    }
+
+    return new_p;
+}

While I can see why having a re-allocation function may be handy,

Quoted 8 lines

explicit / direct use of _xmalloc() and _xzalloc() are discouraged, in favor of the more type-safe underscore-less variants. I can't see though how a type-safe "realloc" could look like, except for arrays. If resizing arrays is all you're after, I'd like to recommend to go that route rather then the suggested one here. If resizing arbitrary objects is the goal, then what you suggest may be the only route, but I'd still be not overly happy to see such added.

I can see 3 uses for realloc:

a. re-allocate generic data buffer
b. re-allocate array
c. re-allocate struct with flexible buffer.

option c. is about structures like this:

struct arrlen

{
         size_t len;
         int data[1];
};

This is Oleksandr's case.

So for option a. we can use _xreallocate(ptr, size, align) For option b. we can use xrealloc_array(_ptr, _type, _num) And for option c. I propose to implement the following macro:

Quoted 8 lines

#define realloc_flex_struct(_ptr, _type, _field, _len) \
((_type *)_xrealloc(_ptr, offsetof(_type, _field[_len]) , __alignof__(_type)))

It can be used in the following way:

newptr = realloc_flex_struct(ptr, struct arrlen, newsize);

As you can see, this approach is type-safe and covers Oleksanrd's case.

This looks fine to me, but then wants to be accompanied by a similar xmalloc_flex_struct(), which could be used right away to replace a number of open-coded instances of the above.

There's one more thing for the re-alloc case though (besides cosmetic aspects): The incoming pointer should also be verified to be of correct type.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#22 lore raw

Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function

Hi, Jan, Volodymyr.

Quoted 21 lines
Quoted 21 lines

  c. re-allocate struct with flexible buffer.

option c. is about structures like this:

struct arrlen
{
         size_t len;
         int data[1];
};

This is Oleksandr's case.

So for option a. we can use _xreallocate(ptr, size, align) For option b. we can use xrealloc_array(_ptr, _type, _num) And for option c. I propose to implement the following macro:

#define realloc_flex_struct(_ptr, _type, _field, _len)                        \
  ((_type *)_xrealloc(_ptr, offsetof(_type, _field[_len]) , __alignof__(_type)))

It can be used in the following way:

Quoted 7 lines
Quoted 3 lines

newptr = realloc_flex_struct(ptr, struct arrlen, newsize);

As you can see, this approach is type-safe and covers Oleksanrd's case.

This looks fine to me, but then wants to be accompanied by a similar xmalloc_flex_struct(), which could be used right away to replace a number of open-coded instances of the above.

Thank you Volodymyr for the idea. Looks like we can get a type-safe approach which looks suitable for my particular case.

So, I need to focus on the proper implementation of non type-safe (_xrealloc) variant in the first place taking into the account Jan's comments. Then I will be back to the suggested type-safe marco (realloc_flex_struct).

Quoted 4 lines

There's one more thing for the re-alloc case though (besides cosmetic aspects): The incoming pointer should also be verified to be of correct type.

Jan, how this could be technically implemented, or are these any existing examples in Xen?

Quoted 2 lines

Jan

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#23 lore raw

Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function

Quoted 5 lines · Oleksandr
Quoted 3 lines

There's one more thing for the re-alloc case though (besides cosmetic aspects): The incoming pointer should also be verified to be of correct type.

Jan, how this could be technically implemented, or are these any existing examples in Xen?

See x86's copy_to_guest_offset(), for example. To get the compiler to emit a warning (at least), a (typically otherwise dead) comparison of pointers is commonly used.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#24 lore raw

Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function

In reply to Oleksandr

(I'm sorry if you receive duplicates of this, but I've got a reply back from our mail system that several of the recipients did not have their host names resolved correctly on the first attempt.)

Quoted 5 lines · Oleksandr
Quoted 3 lines

There's one more thing for the re-alloc case though (besides cosmetic aspects): The incoming pointer should also be verified to be of correct type.

Jan, how this could be technically implemented, or are these any existing examples in Xen?

See x86's copy_to_guest_offset(), for example. To get the compiler to emit a warning (at least), a (typically otherwise dead) comparison of pointers is commonly used.

Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#25 lore raw

Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function

On 08.08.19 10:05, Jan Beulich wrote:

Hi Jan

Quoted 3 lines

(I'm sorry if you receive duplicates of this, but I've got a reply back from our mail system that several of the recipients did not have their host names resolved correctly on the first attempt.)

Absolutely no problem.

Quoted 4 lines
Quoted 1 line

Jan, how this could be technically implemented, or are these any existing examples in Xen?

See x86's copy_to_guest_offset(), for example. To get the compiler to emit a warning (at least), a (typically otherwise dead) comparison of pointers is commonly used.

Thank you for the pointer. It is clear now.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#26 lore raw

PATCH v2 4/6 iommu/arm: Add lightweight iommu_fwspec support

In reply to Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

We need to have some abstract way to add new device to the IOMMU based on the generic IOMMU DT binding [1] which can be used for both DT (right now) and ACPI (in future).

For that reason we can borrow the idea used in Linux these days called "iommu_fwspec". Having this in, it will be possible to configure IOMMU master interfaces of the device (device IDs) from a single common place and avoid keeping almost identifical look-up implementations in each IOMMU driver.

There is no need to port the whole implementation of "iommu_fwspec" to Xen, we could, probably, end up with a much simpler solution, some "stripped down" version which fits our requirments.

So, this patch adds the following:
1. A common structure "iommu_fwspec" to hold the the per-device firmware data
2. New member "iommu_fwspec" of struct device
3. Functions/helpers to deal with "dev->iommu_fwspec"

It should be noted that in comparing with original "iommu_fwspec" Xen's variant doesn't contain some fields, which are not really needed at the moment (ops, flag) and "iommu_fwnode" field was replaced by "iommu_dev" to avoid porting a lot of code (to support "fwnode_handle") with little benefit.

Next patch in this series will make use of that support.

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt

---
 xen/drivers/passthrough/arm/Makefile       |  2 +-
 xen/drivers/passthrough/arm/iommu_fwspec.c | 91 ++++++++++++++++++++++++++++++
 xen/include/asm-arm/device.h               |  1 +
 xen/include/asm-arm/iommu.h                |  2 +
 xen/include/asm-arm/iommu_fwspec.h         | 65 +++++++++++++++++++++
 5 files changed, 160 insertions(+), 1 deletion(-)
 create mode 100644 xen/drivers/passthrough/arm/iommu_fwspec.c
 create mode 100644 xen/include/asm-arm/iommu_fwspec.h
xen/drivers/passthrough/arm/Makefile +1 -1
index 4abb87a..5fbad45 100644
@@ -1,2 +1,2 @@
-obj-y += iommu.o iommu_helpers.o
+obj-y += iommu.o iommu_helpers.o iommu_fwspec.o
 obj-$(CONFIG_ARM_SMMU) += smmu.o
xen/drivers/passthrough/arm/iommu_fwspec.c new file +91 -0

91 lines not on this page · raw message

xen/include/asm-arm/device.h +1 -0
index ee1c3bc..ee7cff2 100644
@@ -18,6 +18,7 @@ struct device
     struct dt_device_node *of_node; /* Used by drivers imported from Linux */
 #endif
     struct dev_archdata archdata;
+    struct iommu_fwspec *iommu_fwspec; /* per-device IOMMU instance data */
 };
 
 typedef struct device device_t;
xen/include/asm-arm/iommu.h +2 -0
index 20d865e..1853bd9 100644
@@ -14,6 +14,8 @@
 #ifndef __ARCH_ARM_IOMMU_H__
 #define __ARCH_ARM_IOMMU_H__
 
+#include <asm/iommu_fwspec.h>
+
 struct arch_iommu
 {
     /* Private information for the IOMMU drivers */
xen/include/asm-arm/iommu_fwspec.h new file +65 -0

65 lines not on this page · raw message

Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#27 lore raw

Re: [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support

Hi Oleksandr,

Quoted 10 lines · Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

We need to have some abstract way to add new device to the IOMMU based on the generic IOMMU DT binding [1] which can be used for both DT (right now) and ACPI (in future).

For that reason we can borrow the idea used in Linux these days called "iommu_fwspec". Having this in, it will be possible to configure IOMMU master interfaces of the device (device IDs) from a single common place and avoid keeping almost identifical look-up

s/identifical/identical/

Quoted 5 lines

implementations in each IOMMU driver.

There is no need to port the whole implementation of "iommu_fwspec" to Xen, we could, probably, end up with a much simpler solution, some "stripped down" version which fits our requirments.

s/requirments/requirements/

Quoted 57 lines

So, this patch adds the following:
1. A common structure "iommu_fwspec" to hold the the per-device firmware data
2. New member "iommu_fwspec" of struct device
3. Functions/helpers to deal with "dev->iommu_fwspec"

It should be noted that in comparing with original "iommu_fwspec" Xen's variant doesn't contain some fields, which are not really needed at the moment (ops, flag) and "iommu_fwnode" field was replaced by "iommu_dev" to avoid porting a lot of code (to support "fwnode_handle") with little benefit.

Next patch in this series will make use of that support.

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt

---
  xen/drivers/passthrough/arm/Makefile       |  2 +-
  xen/drivers/passthrough/arm/iommu_fwspec.c | 91 ++++++++++++++++++++++++++++++
  xen/include/asm-arm/device.h               |  1 +
  xen/include/asm-arm/iommu.h                |  2 +
  xen/include/asm-arm/iommu_fwspec.h         | 65 +++++++++++++++++++++
  5 files changed, 160 insertions(+), 1 deletion(-)

create mode 100644 xen/drivers/passthrough/arm/iommu_fwspec.c create mode 100644 xen/include/asm-arm/iommu_fwspec.h

xen/drivers/passthrough/arm/Makefile +1 -1
index 4abb87a..5fbad45 100644
@@ -1,2 +1,2 @@
-obj-y += iommu.o iommu_helpers.o
+obj-y += iommu.o iommu_helpers.o iommu_fwspec.o
  obj-$(CONFIG_ARM_SMMU) += smmu.o
xen/drivers/passthrough/arm/iommu_fwspec.c new file +15 -0
new file mode 100644
index 0000000..3474192
@@ -0,0 +1,91 @@
+/*
+ * xen/drivers/passthrough/arm/iommu_fwspec.c
+ *
+ * Contains functions to maintain per-device firmware data
+ *
+ * Based on Linux's iommu_fwspec support you can find at:
+ *    drivers/iommu/iommu.c
+ *
+ * Copyright (C) 2019 EPAM Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,

xen/drivers/passthrough/arm/iommu_fwspec.c line 25

Quoted 10 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +10 -0
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/lib.h>
+#include <xen/iommu.h>

Please order the headers alphabetically.

NIT: Can you a newline between xen and asm headers?

Quoted 16 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +15 -0
+#include <asm/device.h>
+#include <asm/iommu_fwspec.h>
 
+
+int iommu_fwspec_init(struct device *dev, struct device *iommu_dev)
+{
+    struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+
+    if ( fwspec )
+        return 0;
+
+    fwspec = xzalloc(struct iommu_fwspec);
+    if ( !fwspec )
+        return -ENOMEM;
+
+    fwspec->iommu_dev = iommu_dev;

xen/drivers/passthrough/arm/iommu_fwspec.c line 50

Quoted 10 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +10 -0
+    dev_iommu_fwspec_set(dev, fwspec);
+
+    return 0;
+}
+
+void iommu_fwspec_free(struct device *dev)
+{
+    struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+
+    if ( fwspec )

xfree is able to deal with NULL pointer, so the check is not necessary.

xen/drivers/passthrough/arm/iommu_fwspec.c line 57

Quoted 7 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +7 -0
+    {
+        xfree(fwspec);
+        dev_iommu_fwspec_set(dev, NULL);
+    }
+}
+
+int iommu_fwspec_add_ids(struct device *dev, uint32_t *ids, int num_ids)

While I realize the prototype is coming from Linux, num_ids cannot be negative (the code below would not work properly). So the parameter should be unsigned.

xen/drivers/passthrough/arm/iommu_fwspec.c line 61

Quoted 4 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +4 -0
+{
+    struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+    size_t size;
+    int i;

Any variable that can't be negative should be unsigned.

Quoted 47 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +30 -0
+
+    if ( !fwspec )
+        return -EINVAL;
+
+    size = offsetof(struct iommu_fwspec, ids[fwspec->num_ids + num_ids]);
+    if ( size > sizeof(*fwspec) )
+    {
+        fwspec = _xrealloc(fwspec, size, sizeof(void *));
+        if ( !fwspec )
+            return -ENOMEM;
+
+        dev_iommu_fwspec_set(dev, fwspec);
+    }
+
+    for ( i = 0; i < num_ids; i++ )
+        fwspec->ids[fwspec->num_ids + i] = ids[i];
+
+    fwspec->num_ids += num_ids;
+
+    return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/include/asm-arm/device.h +1 -0
index ee1c3bc..ee7cff2 100644
@@ -18,6 +18,7 @@ struct device
      struct dt_device_node *of_node; /* Used by drivers imported from Linux */
  #endif
      struct dev_archdata archdata;
+    struct iommu_fwspec *iommu_fwspec; /* per-device IOMMU instance data */
  };
 
  typedef struct device device_t;
xen/include/asm-arm/iommu.h +0 -0
index 20d865e..1853bd9 100644

xen/include/asm-arm/iommu.h line 17

Quoted 4 lines
xen/include/asm-arm/iommu.h +1 -0
@@ -14,6 +14,8 @@
  #ifndef __ARCH_ARM_IOMMU_H__
  #define __ARCH_ARM_IOMMU_H__
 
+#include <asm/iommu_fwspec.h>

iommu.h does not seem to use anything defined in iommu_fwspec.h. So why do you include it here?

Quoted 30 lines
xen/include/asm-arm/iommu.h +1 -0
+
  struct arch_iommu
  {
      /* Private information for the IOMMU drivers */
xen/include/asm-arm/iommu_fwspec.h new file +20 -0
new file mode 100644
index 0000000..0676285
@@ -0,0 +1,65 @@
+/*
+ * xen/include/asm-arm/iommu_fwspec.h
+ *
+ * Contains a common structure to hold the per-device firmware data and
+ * declaration of functions used to maintain that data
+ *
+ * Based on Linux's iommu_fwspec support you can find at:
+ *    include/linux/iommu.h
+ *
+ * Copyright (C) 2019 EPAM Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *

xen/include/asm-arm/iommu_fwspec.h line 30

Quoted 10 lines
xen/include/asm-arm/iommu_fwspec.h new file +10 -0
+ * You should have received a copy of the GNU General Public
+ * License along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __ARCH_ARM_IOMMU_FWSPEC_H__
+#define __ARCH_ARM_IOMMU_FWSPEC_H__
+
+/* per-device IOMMU instance data */
+struct iommu_fwspec {
+    /* device which represents this IOMMU H/W */

Did you intend to say "this device's IOMMU"?

Quoted 26 lines
xen/include/asm-arm/iommu_fwspec.h new file +25 -0
+    struct device *iommu_dev;
+    /* IOMMU driver private data for this device */
+    void *iommu_priv;
+    /* number of associated device IDs */
+    unsigned int num_ids;
+    /* IDs which this device may present to the IOMMU */
+    uint32_t ids[1];
+};
+
+int iommu_fwspec_init(struct device *dev, struct device *iommu_dev);
+void iommu_fwspec_free(struct device *dev);
+int iommu_fwspec_add_ids(struct device *dev, uint32_t *ids, int num_ids);
+
+static inline struct iommu_fwspec *dev_iommu_fwspec_get(struct device *dev)
+{
+    return dev->iommu_fwspec;
+}
+
+static inline void dev_iommu_fwspec_set(struct device *dev,
+                                        struct iommu_fwspec *fwspec)
+{
+    dev->iommu_fwspec = fwspec;
+}
+
+#endif /* __ARCH_ARM_IOMMU_FWSPEC_H__ */

xen/include/asm-arm/iommu_fwspec.h line 65

Quoted 10 lines
xen/include/asm-arm/iommu_fwspec.h new file +10 -0
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#28 lore raw

Re: [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support

Quoted 1 line · Julien Grall

Hi Oleksandr,

Hi Julien.

Quoted 8 lines · Oleksandr Tyshchenko
Quoted 6 lines

From: Oleksandr Tyshchenko <[email protected]>

We need to have some abstract way to add new device to the IOMMU based on the generic IOMMU DT binding [1] which can be used for both DT (right now) and ACPI (in future).

Quoted 6 lines
Quoted 4 lines

For that reason we can borrow the idea used in Linux these days called "iommu_fwspec". Having this in, it will be possible to configure IOMMU master interfaces of the device (device IDs) from a single common place and avoid keeping almost identifical look-up

s/identifical/identical/

ok

Quoted 8 lines
Quoted 6 lines

implementations in each IOMMU driver.

There is no need to port the whole implementation of "iommu_fwspec" to Xen, we could, probably, end up with a much simpler solution, some "stripped down" version which fits our requirments.

s/requirments/requirements/

ok

Quoted 9 lines

xen/drivers/passthrough/arm/iommu_fwspec.c line 25

Quoted 5 lines

+ */

xen/drivers/passthrough/arm/iommu_fwspec.c new file +3 -0
+
+#include <xen/lib.h>
+#include <xen/iommu.h>

Please order the headers alphabetically.

NIT: Can you a newline between xen and asm headers?

Will do

Quoted 21 lines
Quoted 21 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +19 -0
+#include <asm/device.h>
+#include <asm/iommu_fwspec.h>
 
+
+int iommu_fwspec_init(struct device *dev, struct device *iommu_dev)
+{
+    struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+
+    if ( fwspec )
+        return 0;
+
+    fwspec = xzalloc(struct iommu_fwspec);
+    if ( !fwspec )
+        return -ENOMEM;
+
+    fwspec->iommu_dev = iommu_dev;
+    dev_iommu_fwspec_set(dev, fwspec);
+
+    return 0;
+}
Quoted 8 lines

xen/drivers/passthrough/arm/iommu_fwspec.c line 50

Quoted 6 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +6 -0
+
+void iommu_fwspec_free(struct device *dev)
+{
+    struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+
+    if ( fwspec )

xfree is able to deal with NULL pointer, so the check is not necessary.

Yes, the reason I left this check is to not perform an extra operation (dev_iommu_fwspec_set). Shall I drop this check anyway?

Quoted 11 lines
Quoted 9 lines

+    {
+        xfree(fwspec);
+        dev_iommu_fwspec_set(dev, NULL);
+    }

xen/drivers/passthrough/arm/iommu_fwspec.c new file +3 -0
+}
+
+int iommu_fwspec_add_ids(struct device *dev, uint32_t *ids, int 

num_ids)

While I realize the prototype is coming from Linux, num_ids cannot be negative (the code below would not work properly). So the parameter should be unsigned.

Agree, will use unsigned.

Quoted 7 lines
Quoted 5 lines

+{
+    struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); +    size_t size;
+    int i;

Any variable that can't be negative should be unsigned.

Yes, will follow.

Quoted 10 lines
Quoted 8 lines
xen/include/asm-arm/iommu.h +0 -0
index 20d865e..1853bd9 100644
@@ -14,6 +14,8 @@
   #ifndef __ARCH_ARM_IOMMU_H__
   #define __ARCH_ARM_IOMMU_H__

  +#include <asm/iommu_fwspec.h>

iommu.h does not seem to use anything defined in iommu_fwspec.h. So why do you include it here?

I was thinking that every source which includes iommu.h will get iommu_fwspec.h included indirectly. No need to include iommu_fwspec.h in many sources.

This was a reason. Shall I included it directly where needed?

Quoted 36 lines
Quoted 36 lines

+
  struct arch_iommu
  {
      /* Private information for the IOMMU drivers */

xen/include/asm-arm/iommu_fwspec.h new file +22 -0
new file mode 100644
index 0000000..0676285
@@ -0,0 +1,65 @@
+/*
+ * xen/include/asm-arm/iommu_fwspec.h
+ *
+ * Contains a common structure to hold the per-device firmware data and
+ * declaration of functions used to maintain that data
+ *
+ * Based on Linux's iommu_fwspec support you can find at:
+ *    include/linux/iommu.h
+ *
+ * Copyright (C) 2019 EPAM Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; If not, see 

<http://www.gnu.org/licenses/>.
+ */
+

Quoted 8 lines

xen/include/asm-arm/iommu_fwspec.h line 30

Quoted 6 lines
xen/include/asm-arm/iommu_fwspec.h new file +6 -0
+#ifndef __ARCH_ARM_IOMMU_FWSPEC_H__
+#define __ARCH_ARM_IOMMU_FWSPEC_H__
+
+/* per-device IOMMU instance data */
+struct iommu_fwspec {
+    /* device which represents this IOMMU H/W */

Did you intend to say "this device's IOMMU"?

Exactly)

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#29 lore raw

Re: [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support

Hi,

Quoted 6 lines · Oleksandr
Quoted 2 lines · Julien Grall

xfree is able to deal with NULL pointer, so the check is not necessary.

Yes, the reason I left this check is to not perform an extra operation (dev_iommu_fwspec_set). Shall I drop this check anyway?

I can't see any issue to do the extra operation. This is not hotpath and it is harmless.

Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
xen/include/asm-arm/iommu.h +0 -0
index 20d865e..1853bd9 100644
@@ -14,6 +14,8 @@
   #ifndef __ARCH_ARM_IOMMU_H__
   #define __ARCH_ARM_IOMMU_H__

  +#include <asm/iommu_fwspec.h>

iommu.h does not seem to use anything defined in iommu_fwspec.h. So why do you include it here?

Quoted 3 lines

I was thinking that every source which includes iommu.h will get iommu_fwspec.h included indirectly. No need to include iommu_fwspec.h in many sources.

This was a reason. Shall I included it directly where needed?

There are a few cases where iommu.h is required but not iommu_fwspec.h. In general, I would prefer if headers are only included when strictly necessary.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#30 lore raw

Re: [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support

Quoted 1 line · Julien Grall

Hi,

Hi Julien

Quoted 10 lines · Oleksandr
Quoted 6 lines · Julien Grall
Quoted 2 lines

xfree is able to deal with NULL pointer, so the check is not necessary.

Yes, the reason I left this check is to not perform an extra operation (dev_iommu_fwspec_set). Shall I drop this check anyway?

I can't see any issue to do the extra operation. This is not hotpath and it is harmless.

ok, will drop.

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 10 lines
xen/include/asm-arm/iommu.h +0 -0
index 20d865e..1853bd9 100644
@@ -14,6 +14,8 @@
   #ifndef __ARCH_ARM_IOMMU_H__
   #define __ARCH_ARM_IOMMU_H__

  +#include <asm/iommu_fwspec.h>

iommu.h does not seem to use anything defined in iommu_fwspec.h. So why do you include it here?

Quoted 5 lines
Quoted 3 lines

I was thinking that every source which includes iommu.h will get iommu_fwspec.h included indirectly. No need to include iommu_fwspec.h in many sources.

This was a reason. Shall I included it directly where needed?

There are a few cases where iommu.h is required but not iommu_fwspec.h. In general, I would prefer if headers are only included when strictly necessary.

got it, will drop from here and include where necessary.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#31 lore raw

Re: [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support

In reply to Oleksandr Tyshchenko

Hi Oleksandr,

One more comment :).

xen/drivers/passthrough/arm/iommu_fwspec.c line 33

Quoted 5 lines · Oleksandr Tyshchenko
xen/drivers/passthrough/arm/iommu_fwspec.c new file +5 -0
+int iommu_fwspec_init(struct device *dev, struct device *iommu_dev)
+{
+    struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+
+    if ( fwspec )

I would actually check the iommu_dev passed in parameter is the same as stored. We expect all device to be protected by only one IOMMU. So better to be safe than allow overriding ;).

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#32 lore raw

Re: [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support

Quoted 1 line · Julien Grall

Hi Oleksandr,

Hi Julien.

Quoted 11 lines

One more comment :).

xen/drivers/passthrough/arm/iommu_fwspec.c line 33

Quoted 5 lines · Oleksandr Tyshchenko
xen/drivers/passthrough/arm/iommu_fwspec.c new file +5 -0
+int iommu_fwspec_init(struct device *dev, struct device *iommu_dev)
+{
+    struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+
+    if ( fwspec )

I would actually check the iommu_dev passed in parameter is the same as stored. We expect all device to be protected by only one IOMMU. So better to be safe than allow overriding ;).

Actually, it makes sense, will add appropriate check.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#33 lore raw

PATCH v2 5/6 iommu/arm: Introduce iommu_add_dt_device API

In reply to Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

This patch adds new iommu_add_dt_device API for adding DT device to the IOMMU using generic IOMMU DT binding [1] and previously added "iommu_fwspec" support.

New function parses the DT binding, prepares "dev->iommu_fwspec" with correct information and calls the IOMMU driver using "add_device" callback to register new DT device.
The IOMMU driver's responsibility is to check whether "dev->iommu_fwspec" is initialized and mark that device as protected.

The additional benefit here is to avoid to go through the whole DT multiple times in IOMMU driver trying to locate master devices which belong to each IOMMU device being probed.

The upcoming IPMMU driver will have "add_device" callback implemented.

I hope, this patch won't break SMMU driver's functionality, which doesn't have this callback implemented.

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt

---
 xen/arch/arm/domain_build.c         | 12 ++++++++++
 xen/drivers/passthrough/arm/iommu.c | 45 +++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/iommu.h         |  3 +++
 3 files changed, 60 insertions(+)
xen/arch/arm/domain_build.c +12 -0
index d983677..d67f7d4 100644
@@ -1241,6 +1241,18 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev,
     u64 addr, size;
     bool need_mapping = !dt_device_for_passthrough(dev);
 
+    if ( dt_parse_phandle(dev, "iommus", 0) )
+    {
+        dt_dprintk("%s add to iommu\n", dt_node_full_name(dev));
+        res = iommu_add_dt_device(dev);
+        if ( res )
+        {
+            printk(XENLOG_ERR "Failed to add %s to the IOMMU\n",
+                   dt_node_full_name(dev));
+            return res;
+        }
+    }
+
     nirq = dt_number_of_irq(dev);
     naddr = dt_number_of_address(dev);
 
xen/drivers/passthrough/arm/iommu.c +45 -0

48 lines not on this page · raw message

xen/include/asm-arm/iommu.h +3 -0
index 1853bd9..06b07fa 100644
@@ -28,6 +28,9 @@ struct arch_iommu
 const struct iommu_ops *iommu_get_ops(void);
 void iommu_set_ops(const struct iommu_ops *ops);
 
+/* helper to add DT device to the IOMMU */
+int iommu_add_dt_device(struct dt_device_node *np);
+
 /* mapping helpers */
 int __must_check arm_iommu_map_page(struct domain *d, dfn_t dfn, mfn_t mfn,
                                     unsigned int flags,
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#34 lore raw

Re: [Xen-devel] [PATCH V2 5/6] iommu/arm: Introduce iommu_add_dt_device API

Hi Oleksandr,

Quoted 12 lines · Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

This patch adds new iommu_add_dt_device API for adding DT device to the IOMMU using generic IOMMU DT binding [1] and previously added "iommu_fwspec" support.

New function parses the DT binding, prepares "dev->iommu_fwspec" with correct information and calls the IOMMU driver using "add_device" callback to register new DT device.
The IOMMU driver's responsibility is to check whether "dev->iommu_fwspec" is initialized and mark that device as protected.

Quoted 8 lines

The additional benefit here is to avoid to go through the whole DT multiple times in IOMMU driver trying to locate master devices which belong to each IOMMU device being probed.

The upcoming IPMMU driver will have "add_device" callback implemented.

I hope, this patch won't break SMMU driver's functionality, which doesn't have this callback implemented.

The last two sentence does not really belong to the commit message. So I think they should go after ---.

Anyway, the only concern for the SMMU is to not break the old bindings. New bindings are not supported, so it does not matter whether they are broken or not. Once this series is merged, we can have a look how new bindings for the SMMU can be supported.

Quoted 15 lines

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt

---
  xen/arch/arm/domain_build.c         | 12 ++++++++++
  xen/drivers/passthrough/arm/iommu.c | 45 +++++++++++++++++++++++++++++++++++++
  xen/include/asm-arm/iommu.h         |  3 +++
  3 files changed, 60 insertions(+)
xen/arch/arm/domain_build.c +0 -0
index d983677..d67f7d4 100644

xen/arch/arm/domain_build.c static int __init handle_device(struct domain *d, struct dt_device_node *dev, · line 1244

Quoted 4 lines
xen/arch/arm/domain_build.c +1 -0
@@ -1241,6 +1241,18 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev,
      u64 addr, size;
      bool need_mapping = !dt_device_for_passthrough(dev);
 
+    if ( dt_parse_phandle(dev, "iommus", 0) )

I don't particularly like this check. dt_parse_phandle is non-trivial to execute.

TBH, what we should do is trying to call iommu_add_dt_device if IOMMU is enabled. We can then return a recognizable value to tell the device is not protected.

Quoted 22 lines
xen/arch/arm/domain_build.c +11 -0
+    {
+        dt_dprintk("%s add to iommu\n", dt_node_full_name(dev));
+        res = iommu_add_dt_device(dev);
+        if ( res )
+        {
+            printk(XENLOG_ERR "Failed to add %s to the IOMMU\n",
+                   dt_node_full_name(dev));
+            return res;
+        }
+    }
+
      nirq = dt_number_of_irq(dev);
      naddr = dt_number_of_address(dev);
 
xen/drivers/passthrough/arm/iommu.c +0 -0
index 3195919..19516af 100644
@@ -113,3 +113,48 @@ int arch_iommu_populate_page_table(struct domain *d)
  void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
  {
  }

xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 125

Quoted 10 lines
xen/drivers/passthrough/arm/iommu.c +10 -0
+
+int __init iommu_add_dt_device(struct dt_device_node *np)
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+    struct dt_phandle_args iommu_spec;
+    struct device *dev = dt_to_dev(np);
+    int rc = 1, index = 0;
+
+    if ( !iommu_enabled || !ops || !ops->add_device )
+        return 0;

While I agree that !iommu_enabled should return 0, for the two others I am not entirely sure this is the right thing to do.

!ops is definitely an error because if you have the IOMMU enabled then you should have ops installed.

!ops->add_device means that you will not be able to add any device using the new bindings. IOW, the device will be unusable later one as most likely the IOMMU was configured to deny any transaction. Therefore, this should return an error.

xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 130

Quoted 5 lines
xen/drivers/passthrough/arm/iommu.c +5 -0
+
+    if ( dev_iommu_fwspec_get(dev) )
+        return -EEXIST;
+
+    /* According to the Documentation/devicetree/bindings/iommu/iommu.txt */

This file does not exist in Xen, so Linux should at least be mentioned in the comment.

xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 141

Quoted 11 lines
xen/drivers/passthrough/arm/iommu.c +11 -0
+    while ( !dt_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+                                        index, &iommu_spec) )
+    {
+        if ( !dt_device_is_available(iommu_spec.np) )
+            break;
+
+        rc = iommu_fwspec_init(dev, &iommu_spec.np->dev);
+        if ( rc )
+            break;
+
+        rc = iommu_fwspec_add_ids(dev, iommu_spec.args, 1);

Here you assume that there will at least always be one cells and the first cell is the IDs. For a first, #iommu-cells may be 0 (and therefore no cells) when the master IOMMU device cannot be configured.

Furthermore, the content of the #iommu-cells depends on the driver. This is why Linux provides a callback of_xlate to let the driver decide how to interpret it.

For instance, the SMMU can support either 1 or 2 cells. It also may need to look-up for other properties in the node (e.g stream-match-mask).

So I think we also want to provide the of_xlate in Xen.

Quoted 25 lines
xen/drivers/passthrough/arm/iommu.c +19 -0
+        if ( rc )
+            break;
+
+        index++;
+    }
+
+    /*
+     * Add DT device to the IOMMU if latter is present and available.
+     * The IOMMU driver's responsibility is to check whether dev->iommu_fwspec
+     * field is initialized and mark that device as protected.
+     */
+    if ( !rc )
+        rc = ops->add_device(0, dev);
+
+    if ( rc < 0 )
+        iommu_fwspec_free(dev);
+
+    return rc < 0 ? rc : 0;
+}
xen/include/asm-arm/iommu.h +0 -0
index 1853bd9..06b07fa 100644

xen/include/asm-arm/iommu.h struct arch_iommu · line 36

Quoted 9 lines
xen/include/asm-arm/iommu.h +3 -0
@@ -28,6 +28,9 @@ struct arch_iommu
  const struct iommu_ops *iommu_get_ops(void);
  void iommu_set_ops(const struct iommu_ops *ops);
 
+/* helper to add DT device to the IOMMU */
+int iommu_add_dt_device(struct dt_device_node *np);
+
  /* mapping helpers */
  int __must_check arm_iommu_map_page(struct domain *d, dfn_t dfn, mfn_t mfn,
                                      unsigned int flags,

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#35 lore raw

Re: [Xen-devel] [PATCH V2 5/6] iommu/arm: Introduce iommu_add_dt_device API

Quoted 1 line · Julien Grall

Hi Oleksandr,

Hi Julien

Quoted 18 lines · Oleksandr Tyshchenko
Quoted 16 lines

From: Oleksandr Tyshchenko <[email protected]>

This patch adds new iommu_add_dt_device API for adding DT device to the IOMMU using generic IOMMU DT binding [1] and previously added "iommu_fwspec" support.

New function parses the DT binding, prepares "dev->iommu_fwspec" with correct information and calls the IOMMU driver using "add_device" callback to register new DT device.
The IOMMU driver's responsibility is to check whether "dev->iommu_fwspec" is initialized and mark that device as protected.

The additional benefit here is to avoid to go through the whole DT multiple times in IOMMU driver trying to locate master devices which belong to each IOMMU device being probed.

Quoted 8 lines
Quoted 4 lines

The upcoming IPMMU driver will have "add_device" callback implemented.

I hope, this patch won't break SMMU driver's functionality, which doesn't have this callback implemented.

The last two sentence does not really belong to the commit message. So I think they should go after ---.

Anyway, the only concern for the SMMU is to not break the old bindings. New bindings are not supported, so it does not matter whether they are broken or not. Once this series is merged, we can have a look how new bindings for the SMMU can be supported.

Sounds reasonable.

Quoted 24 lines
Quoted 17 lines

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt

---
  xen/arch/arm/domain_build.c         | 12 ++++++++++ xen/drivers/passthrough/arm/iommu.c | 45
+++++++++++++++++++++++++++++++++++++
  xen/include/asm-arm/iommu.h         |  3 +++

  3 files changed, 60 insertions(+)
xen/arch/arm/domain_build.c +0 -0
index d983677..d67f7d4 100644

xen/arch/arm/domain_build.c static int __init handle_device(struct domain · line 1244

Quoted 4 lines
xen/arch/arm/domain_build.c +0 -0
@@ -1241,6 +1241,18 @@ static int __init handle_device(struct domain 
 *d, struct dt_device_node *dev,
       u64 addr, size;
       bool need_mapping = !dt_device_for_passthrough(dev);
   +    if ( dt_parse_phandle(dev, "iommus", 0) )

I don't particularly like this check. dt_parse_phandle is non-trivial to execute.

Quoted 1 line

TBH, what we should do is trying to call iommu_add_dt_device if IOMMU is enabled. We can then return a recognizable value to tell the device is not protected.

Well. Don't really mind.

Quoted 36 lines
Quoted 23 lines
xen/arch/arm/domain_build.c +11 -0
+    {
+        dt_dprintk("%s add to iommu\n", dt_node_full_name(dev));
+        res = iommu_add_dt_device(dev);
+        if ( res )
+        {
+            printk(XENLOG_ERR "Failed to add %s to the IOMMU\n",
+                   dt_node_full_name(dev));
+            return res;
+        }
+    }
+
       nirq = dt_number_of_irq(dev);
       naddr = dt_number_of_address(dev);

  diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c index 3195919..19516af 100644

xen/drivers/passthrough/arm/iommu.c +0 -0
@@ -113,3 +113,48 @@ int arch_iommu_populate_page_table(struct domain 
 *d)
   void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
   {

  }

xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 125

Quoted 10 lines
xen/drivers/passthrough/arm/iommu.c +10 -0
+
+int __init iommu_add_dt_device(struct dt_device_node *np)
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+    struct dt_phandle_args iommu_spec;
+    struct device *dev = dt_to_dev(np);
+    int rc = 1, index = 0;
+
+    if ( !iommu_enabled || !ops || !ops->add_device )
+        return 0;

While I agree that !iommu_enabled should return 0, for the two others I am not entirely sure this is the right thing to do.

Quoted 1 line

!ops is definitely an error because if you have the IOMMU enabled then you should have ops installed.

Agree.

Quoted 3 lines

!ops->add_device means that you will not be able to add any device using the new bindings. IOW, the device will be unusable later one as most likely the IOMMU was configured to deny any transaction. Therefore, this should return an error.

The initial reason *was* to not break SMMU which hasn't had add_device callback implemented yet. But, I got your point regarding SMMU above (the only concern for the SMMU is to not break the old bindings), so agree here.

Quoted 9 lines
Quoted 7 lines
xen/drivers/passthrough/arm/iommu.c +5 -0
+
+    if ( dev_iommu_fwspec_get(dev) )
+        return -EEXIST;
+
+    /* According to the 

Documentation/devicetree/bindings/iommu/iommu.txt */

This file does not exist in Xen, so Linux should at least be mentioned in the comment.

Will do.

Quoted 15 lines

xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 141

Quoted 12 lines

+    while ( !dt_parse_phandle_with_args(np, "iommus", "#iommu-cells", +                                        index, &iommu_spec) ) +    {
+        if ( !dt_device_is_available(iommu_spec.np) )
+            break;

xen/drivers/passthrough/arm/iommu.c +6 -0
+
+        rc = iommu_fwspec_init(dev, &iommu_spec.np->dev);
+        if ( rc )
+            break;
+
+        rc = iommu_fwspec_add_ids(dev, iommu_spec.args, 1);

Here you assume that there will at least always be one cells and the first cell is the IDs. For a first, #iommu-cells may be 0 (and therefore no cells) when the master IOMMU device cannot be configured.

Quoted 5 lines

Furthermore, the content of the #iommu-cells depends on the driver. This is why Linux provides a callback of_xlate to let the driver decide how to interpret it.

For instance, the SMMU can support either 1 or 2 cells. It also may need to look-up for other properties in the node (e.g stream-match-mask).

So I think we also want to provide the of_xlate in Xen.

Hmm, I was thinking how to end up with only one callback re-used (add_device), really didn't want to add a new one (of_xlate). But, I didn't take into the account that this stuff is a really driver-depended. So, likely yes, we need to provide of_xlate callback.

May I ask some questions to clarify:

1. Do you want me to introduce of_xlate callback in a separate patch (under CONFIG_ARM?)?

2. Can we avoid introducing new API for that callback? iommu_add_dt_device will be able to call it directly.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#36 lore raw

Re: [Xen-devel] [PATCH V2 5/6] iommu/arm: Introduce iommu_add_dt_device API

Hi,

Quoted 8 lines · Oleksandr
Quoted 1 line · Julien Grall

On 8/2/19 5:39 PM, Oleksandr Tyshchenko wrote:

Hmm, I was thinking how to end up with only one callback re-used (add_device), really didn't want to add a new one (of_xlate). But, I didn't take into the account that this stuff is a really driver-depended. So, likely yes, we need to provide of_xlate callback.

May I ask some questions to clarify:

1. Do you want me to introduce of_xlate callback in a separate patch (under CONFIG_ARM?)?

Preferably yes. I think this wants to be under CONFIG_HAS_DEVICE_TREE rather than CONFIG_ARM.

Quoted 1 line

2. Can we avoid introducing new API for that callback?

Do you mean a wrapper for the callback? If so, yes.

Quoted 1 line

iommu_add_dt_device will be able to call it directly.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#37 lore raw

PATCH v2 6/6 iommu/arm: Add Renesas IPMMU-VMSA support

In reply to Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.

Please note, current driver is supposed to work only with newest Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).

The major differences compare to the Linux driver are:

1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.

So, Xen driver is supposed to be used with newest Gen3 SoC revisions only (H3 ES3.0, M3 ES3.0, etc.) which IPMMU H/W supports stage 2 translation table format.

2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.

3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.

CC Julien Grall <[email protected]>
CC Yoshihiro Shimoda <[email protected]>

---
Changes V1 -> V2:
- rewrited driver to use iommu_fwspec
- removed DT parsing code for micro-TLBs
- removed struct ipmmu_vmsa_master_cfg, dev_archdata macro
- added ipmmu_find_mmu_by_dev(), various helpers to access fwspec->iommu_priv
- implemented new callback "add_device" to add master device to IPMMU
- removed ipmmu_protect_masters()
- removed code to locate Root device in the first place, used EAGAIN to request deferred probing
- used printk_once for the system wide error messages in ipmmu_init() which don't need to be shown for every device being probed
- removed map_page/unmap_page implementation, reused them from iommu_helpers.c
- used %pd for printing domaid id
- performed various cosmetic fixes
- changed u32 -> uint32_t, u64 -> uint64_t,
unsigned int -> uint32_t where needed
- clarified TODOs
- clafiried supported SoC versions in config IPMMU_VMSA, set default to "n"
- updated comments in code, provided more accurate description, added new comments where needed
- updated patch description by providing differences between Linux/Xen implementations
- removed fields for cache snoop transaction when configuring IMTTBCR (update from Renesas BSP)

---
 xen/arch/arm/platforms/Kconfig           |    1 +
 xen/drivers/passthrough/Kconfig          |   13 +
 xen/drivers/passthrough/arm/Makefile     |    1 +
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 1342 ++++++++++++++++++++++++++++++
 4 files changed, 1357 insertions(+)
 create mode 100644 xen/drivers/passthrough/arm/ipmmu-vmsa.c
xen/arch/arm/platforms/Kconfig +1 -0
index bc0e9cd..c93a6b2 100644
@@ -25,6 +25,7 @@ config RCAR3
 	bool "Renesas RCar3 support"
 	depends on ARM_64
 	select HAS_SCIF
+	select IPMMU_VMSA
 	---help---
 	Enable all the required drivers for Renesas RCar3
 
xen/drivers/passthrough/Kconfig +13 -0
index a3c0649..3daee16 100644
@@ -12,4 +12,17 @@ config ARM_SMMU
 
 	  Say Y here if your SoC includes an IOMMU device implementing the
 	  ARM SMMU architecture.
+
+config IPMMU_VMSA
+	bool "Renesas IPMMU-VMSA found in R-Car Gen3 SoCs"
+	default n
+	depends on ARM_64
+	---help---
+	  Support for implementations of the Renesas IPMMU-VMSA found
+	  in R-Car Gen3 SoCs.
+
+	  Say Y here if you are using newest R-Car Gen3 SoCs revisions
+	  (H3 ES3.0, M3 ES3.0, etc) which IPMMU hardware supports stage 2
+	  translation table format and is able to use CPU's P2M table as is.
+
 endif
xen/drivers/passthrough/arm/Makefile +1 -0
index 5fbad45..fcd918e 100644
@@ -1,2 +1,3 @@
 obj-y += iommu.o iommu_helpers.o iommu_fwspec.o
 obj-$(CONFIG_ARM_SMMU) += smmu.o
+obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +1342 -0

1,342 lines not on this page · raw message

Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#38 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Hi Oleksandr-san,

I can access the datasheet of this hardware, so that I reviewed this patch. I'm not familar about Xen development rulus, so that some comments might be not good fit. If so, please ignore :)

Quoted 10 lines

From: Oleksandr Tyshchenko, Sent: Saturday, August 3, 2019 1:40 AM

From: Oleksandr Tyshchenko <[email protected]>

The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.

Please note, current driver is supposed to work only with newest Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation

This should be "R-Car Gen3 SoCs", instead of "Gen3 SoCs".

Quoted 5 lines

table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).

The major differences compare to the Linux driver are:

Quoted 7 lines

1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.

So, Xen driver is supposed to be used with newest Gen3 SoC revisions only

Same here.

Quoted 1 line

(H3 ES3.0, M3 ES3.0, etc.) which IPMMU H/W supports stage 2 translation

According to the latest manual, M3 ES3.0 is named as "M3-W+".

Quoted 1 line

table format.

<snip>

Quoted 11 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5 -0
new file mode 100644
index 0000000..a34a8f8
@@ -0,0 +1,1342 @@
+/*
+ * xen/drivers/passthrough/arm/ipmmu-vmsa.c
+ *
+ * Driver for the Renesas IPMMU-VMSA found in R-Car Gen3 SoCs.
+ *

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 15

Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10 -0
+ * The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU)
+ * which provides address translation and access protection functionalities
+ * to processing units and interconnect networks.
+ *
+ * Please note, current driver is supposed to work only with newest Gen3 SoCs
+ * revisions which IPMMU hardware supports stage 2 translation table format and
+ * is able to use CPU's P2M table as is.
+ *
+ * Based on Linux's IPMMU-VMSA driver from Renesas BSP:
+ *    drivers/iommu/ipmmu-vmsa.c

So, I think the Linux's Copyrights should be described here.

Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10 -0
+ * you can found at:
+ *    url: git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git
+ *    branch: v4.14.75-ltsi/rcar-3.9.6
+ *    commit: e206eb5b81a60e64c35fbc3a999b1a0db2b98044
+ * and Xen's SMMU driver:
+ *    xen/drivers/passthrough/arm/smmu.c
+ *
+ * Copyright (C) 2016-2019 EPAM Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 35

Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10 -0
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; If not, see <http://www.gnu.org/licenses/>.

I don't know that Xen license description rule, but since a few source files have SPDX-License-Identifier, can we also use it on the driver?

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 42

Quoted 7 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7 -0
+ */
+
+#include <xen/delay.h>
+#include <xen/err.h>
+#include <xen/irq.h>
+#include <xen/lib.h>
+#include <xen/list.h>

I don't know that Xen passthrough driver rule though, doesn't here need #include <xen/iommu.h>? (The xen/sched.h seems to have it so that no compile error happens though.)

<snip>

Quoted 33 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +33 -0
+/* Registers Definition */
+#define IM_CTX_SIZE    0x40
+
+#define IMCTR                0x0000
+/*
+ * These fields are implemented in IPMMU-MM only. So, can be set for
+ * Root IPMMU only.
+ */
+#define IMCTR_VA64           (1 << 29)
+#define IMCTR_TRE            (1 << 17)
+#define IMCTR_AFE            (1 << 16)
+#define IMCTR_RTSEL_MASK     (3 << 4)
+#define IMCTR_RTSEL_SHIFT    4
+#define IMCTR_TREN           (1 << 3)
+/*
+ * These fields are common for all IPMMU devices. So, can be set for
+ * Cache IPMMUs as well.
+ */
+#define IMCTR_INTEN          (1 << 2)
+#define IMCTR_FLUSH          (1 << 1)
+#define IMCTR_MMUEN          (1 << 0)
+#define IMCTR_COMMON_MASK    (7 << 0)
+
+#define IMCAAR               0x0004
+
+#define IMTTBCR                        0x0008
+#define IMTTBCR_EAE                    (1 << 31)
+#define IMTTBCR_PMB                    (1 << 30)
+#define IMTTBCR_SH1_NON_SHAREABLE      (0 << 28)
+#define IMTTBCR_SH1_OUTER_SHAREABLE    (2 << 28)
+#define IMTTBCR_SH1_INNER_SHAREABLE    (3 << 28)
+#define IMTTBCR_SH1_MASK               (3 << 28)
+#define IMTTBCR_ORGN1_NC               (0 << 26)

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 201

Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10 -0
+#define IMTTBCR_ORGN1_WB_WA            (1 << 26)
+#define IMTTBCR_ORGN1_WT               (2 << 26)
+#define IMTTBCR_ORGN1_WB               (3 << 26)
+#define IMTTBCR_ORGN1_MASK             (3 << 26)
+#define IMTTBCR_IRGN1_NC               (0 << 24)
+#define IMTTBCR_IRGN1_WB_WA            (1 << 24)
+#define IMTTBCR_IRGN1_WT               (2 << 24)
+#define IMTTBCR_IRGN1_WB               (3 << 24)
+#define IMTTBCR_IRGN1_MASK             (3 << 24)
+#define IMTTBCR_TSZ1_MASK              (1f << 16)

At the moment, no one uses it though, this should be (0x1f << 16).

<snip>
+/* Xen IOMMU ops */

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 811

Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8 -0
+static int __must_check ipmmu_iotlb_flush_all(struct domain *d)
+{
+    struct ipmmu_vmsa_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
+
+    if ( !xen_domain || !xen_domain->root_domain )
+        return 0;
+
+    spin_lock(&xen_domain->lock);

Is local irq is already disabled here?
If no, you should use spin_lock_irqsave() because the ipmmu_irq() also gets the lock.
# To be honest, in normal case, any irq on the current implementation

# should not happen though.
Quoted 6 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6 -0
+    ipmmu_tlb_invalidate(xen_domain->root_domain);
+    spin_unlock(&xen_domain->lock);
+
+    return 0;
+}
+

<snip>

Quoted 4 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +4 -0
+static int ipmmu_assign_device(struct domain *d, u8 devfn, struct device *dev,
+                               uint32_t flag)
+{
+    struct ipmmu_vmsa_xen_domain *xen_domain = dom_iommu(d)->arch.priv;

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 936

Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10 -0
+    struct ipmmu_vmsa_domain *domain;
+    int ret;
+
+    if ( !xen_domain )
+        return -EINVAL;
+
+    if ( !to_ipmmu(dev) )
+        return -ENODEV;
+
+    spin_lock(&xen_domain->lock);

Same here.

Quoted 71 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +71 -0
+    /*
+     * The IPMMU context for the Xen domain is not allocated beforehand
+     * (at the Xen domain creation time), but on demand only, when the first
+     * master device being attached to it.
+     * Create Root IPMMU domain which context will be mapped to this Xen domain
+     * if not exits yet.
+     */
+    if ( !xen_domain->root_domain )
+    {
+        domain = ipmmu_alloc_root_domain(d);
+        if ( IS_ERR(domain) )
+        {
+            ret = PTR_ERR(domain);
+            goto out;
+        }
+
+        xen_domain->root_domain = domain;
+    }
+
+    if ( to_domain(dev) )
+    {
+        dev_err(dev, "Already attached to IPMMU domain\n");
+        ret = -EEXIST;
+        goto out;
+    }
+
+    /*
+     * Master devices behind the same Cache IPMMU can be attached to the same
+     * Cache IPMMU domain.
+     * Before creating new IPMMU domain check to see if the required one
+     * already exists for this Xen domain.
+     */
+    domain = ipmmu_get_cache_domain(d, dev);
+    if ( !domain )
+    {
+        /* Create new IPMMU domain this master device will be attached to. */
+        domain = ipmmu_alloc_cache_domain(d);
+        if ( IS_ERR(domain) )
+        {
+            ret = PTR_ERR(domain);
+            goto out;
+        }
+
+        /* Chain new IPMMU domain to the Xen domain. */
+        list_add(&domain->list, &xen_domain->cache_domains);
+    }
+
+    ret = ipmmu_attach_device(domain, dev);
+    if ( ret )
+    {
+        /*
+         * Destroy Cache IPMMU domain only if there are no master devices
+         * attached to it.
+         */
+        if ( !domain->refcount )
+            ipmmu_free_cache_domain(domain);
+    }
+    else
+    {
+        domain->refcount++;
+        set_domain(dev, domain);
+    }
+
+out:
+    spin_unlock(&xen_domain->lock);
+
+    return ret;
+}
+
+static int ipmmu_deassign_device(struct domain *d, struct device *dev)
+{

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1018

Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10 -0
+    struct ipmmu_vmsa_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
+    struct ipmmu_vmsa_domain *domain = to_domain(dev);
+
+    if ( !domain || domain->d != d )
+    {
+        dev_err(dev, "Not attached to %pd\n", d);
+        return -ESRCH;
+    }
+
+    spin_lock(&xen_domain->lock);

Same here.

Quoted 15 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +15 -0
+    ipmmu_detach_device(domain, dev);
+    set_domain(dev, NULL);
+    domain->refcount--;
+
+    /*
+     * Destroy Cache IPMMU domain only if there are no master devices
+     * attached to it.
+     */
+    if ( !domain->refcount )
+        ipmmu_free_cache_domain(domain);
+
+    spin_unlock(&xen_domain->lock);
+
+    return 0;
+}

<snip>

Quoted 11 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +11 -0
+static void __hwdom_init ipmmu_iommu_hwdom_init(struct domain *d)
+{
+    /* Set to false options not supported on ARM. */
+    if ( iommu_hwdom_inclusive )
+        printk(XENLOG_WARNING "ipmmu: map-inclusive dom0-iommu option is not supported on ARM\n");
+    iommu_hwdom_inclusive = false;
+    if ( iommu_hwdom_reserved == 1 )
+        printk(XENLOG_WARNING "ipmmu: map-reserved dom0-iommu option is not supported on ARM\n");
+    iommu_hwdom_reserved = 0;
+
+    arch_iommu_hwdom_init(d);

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1154

Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10 -0
+}
+
+static void ipmmu_iommu_domain_teardown(struct domain *d)
+{
+    struct ipmmu_vmsa_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
+
+    if ( !xen_domain )
+        return;
+
+    spin_lock(&xen_domain->lock);

Same here.

Quoted 5 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5 -0
+    /*
+     * Destroy Root IPMMU domain which context is mapped to this Xen domain
+     * if exits.
+     */
+    if ( xen_domain->root_domain )

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1169

Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10 -0
+        ipmmu_free_root_domain(xen_domain->root_domain);
+
+    spin_unlock(&xen_domain->lock);
+
+    /*
+     * We assume that all master devices have already been detached from
+     * this Xen domain and there must be no associated Cache IPMMU domains
+     * in use.
+     */
+    ASSERT(list_empty(&xen_domain->cache_domains));

I think this should be in the spin lock held by &xen_domain->lock.

Quoted 9 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +9 -0
+    xfree(xen_domain);
+    dom_iommu(d)->arch.priv = NULL;
+}
+
+static const struct iommu_ops ipmmu_iommu_ops =
+{
+    .init            = ipmmu_iommu_domain_init,
+    .hwdom_init      = ipmmu_iommu_hwdom_init,
+    .teardown        = ipmmu_iommu_domain_teardown,

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1188

Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10 -0
+    .iotlb_flush     = ipmmu_iotlb_flush,
+    .iotlb_flush_all = ipmmu_iotlb_flush_all,
+    .assign_device   = ipmmu_assign_device,
+    .reassign_device = ipmmu_reassign_device,
+    .map_page        = arm_iommu_map_page,
+    .unmap_page      = arm_iommu_unmap_page,
+    .add_device      = ipmmu_add_device,
+};
+
+/* RCAR GEN3 product and cut information. */

"R-Car Gen3 SoCs" is better than "RCAR GEN3".

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1191

Quoted 3 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +3 -0
+#define RCAR_PRODUCT_MASK    0x00007F00
+#define RCAR_PRODUCT_H3      0x00004F00
+#define RCAR_PRODUCT_M3      0x00005200

At least, I think we should be M3W, instead of M3. # FYI, M3-W and M3-W+ are the same value.

<snip>

Best regards,
Yoshihiro Shimoda

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#39 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Hi, Shimoda-san.

Thank you for the review.

Quoted 5 lines
Quoted 5 lines

From: Oleksandr Tyshchenko, Sent: Saturday, August 3, 2019 1:40 AM

From: Oleksandr Tyshchenko <[email protected]>

Quoted 7 lines
Quoted 6 lines

The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.

Please note, current driver is supposed to work only with newest Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation

This should be "R-Car Gen3 SoCs", instead of "Gen3 SoCs".

Will update.

Quoted 6 lines
Quoted 6 lines

table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).

The major differences compare to the Linux driver are:

Quoted 8 lines
Quoted 7 lines

1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.

So, Xen driver is supposed to be used with newest Gen3 SoC revisions only

Same here.

ok

Quoted 3 lines
Quoted 2 lines

(H3 ES3.0, M3 ES3.0, etc.) which IPMMU H/W supports stage 2 translation

According to the latest manual, M3 ES3.0 is named as "M3-W+".

Will update.

Quoted 14 lines
Quoted 14 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8 -0
new file mode 100644
index 0000000..a34a8f8
@@ -0,0 +1,1342 @@
+/*
+ * xen/drivers/passthrough/arm/ipmmu-vmsa.c
+ *
+ * Driver for the Renesas IPMMU-VMSA found in R-Car Gen3 SoCs.
+ *
+ * The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU)
+ * which provides address translation and access protection functionalities
+ * to processing units and interconnect networks.
Quoted 8 lines

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 15

Quoted 7 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7 -0
+ *
+ * Please note, current driver is supposed to work only with newest Gen3 SoCs
+ * revisions which IPMMU hardware supports stage 2 translation table format and
+ * is able to use CPU's P2M table as is.
+ *
+ * Based on Linux's IPMMU-VMSA driver from Renesas BSP:
+ *    drivers/iommu/ipmmu-vmsa.c

So, I think the Linux's Copyrights should be described here.

Yes, will add.

Quoted 21 lines
Quoted 11 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10 -0
+ * you can found at:
+ *    url: git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git
+ *    branch: v4.14.75-ltsi/rcar-3.9.6
+ *    commit: e206eb5b81a60e64c35fbc3a999b1a0db2b98044
+ * and Xen's SMMU driver:
+ *    xen/drivers/passthrough/arm/smmu.c
+ *
+ * Copyright (C) 2016-2019 EPAM Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 35

Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10 -0
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; If not, see <http://www.gnu.org/licenses/>.
Quoted 2 lines

I don't know that Xen license description rule, but since a few source files have SPDX-License-Identifier, can we also use it on the driver?

I am afraid, I don't know a correct answer for this question. I would leave this to maintainers.

I just followed sample copyright notice for GPL v2 License according to the document:

http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=CONTRIBUTING

Quoted 10 lines

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 42

Quoted 7 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7 -0
+ */
+
+#include <xen/delay.h>
+#include <xen/err.h>
+#include <xen/irq.h>
+#include <xen/lib.h>
+#include <xen/list.h>

I don't know that Xen passthrough driver rule though, doesn't here need #include <xen/iommu.h>? (The xen/sched.h seems to have it so that no compile error happens though.)

Probably, yes, I should have included that header.

Quoted 12 lines
Quoted 12 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +12 -0
+
+#define IMTTBCR                        0x0008
+#define IMTTBCR_EAE                    (1 << 31)
+#define IMTTBCR_PMB                    (1 << 30)
+#define IMTTBCR_SH1_NON_SHAREABLE      (0 << 28)
+#define IMTTBCR_SH1_OUTER_SHAREABLE    (2 << 28)
+#define IMTTBCR_SH1_INNER_SHAREABLE    (3 << 28)
+#define IMTTBCR_SH1_MASK               (3 << 28)
+#define IMTTBCR_ORGN1_NC               (0 << 26)
+#define IMTTBCR_ORGN1_WB_WA            (1 << 26)
+#define IMTTBCR_ORGN1_WT               (2 << 26)
+#define IMTTBCR_ORGN1_WB               (3 << 26)
Quoted 8 lines

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 201

Quoted 7 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7 -0
+#define IMTTBCR_ORGN1_MASK             (3 << 26)
+#define IMTTBCR_IRGN1_NC               (0 << 24)
+#define IMTTBCR_IRGN1_WB_WA            (1 << 24)
+#define IMTTBCR_IRGN1_WT               (2 << 24)
+#define IMTTBCR_IRGN1_WB               (3 << 24)
+#define IMTTBCR_IRGN1_MASK             (3 << 24)
+#define IMTTBCR_TSZ1_MASK              (1f << 16)

At the moment, no one uses it though, this should be (0x1f << 16).

Will correct.

Quoted 14 lines

<snip>
+/* Xen IOMMU ops */

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 811

Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8 -0
+static int __must_check ipmmu_iotlb_flush_all(struct domain *d)
+{
+    struct ipmmu_vmsa_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
+
+    if ( !xen_domain || !xen_domain->root_domain )
+        return 0;
+
+    spin_lock(&xen_domain->lock);

Is local irq is already disabled here?
If no, you should use spin_lock_irqsave() because the ipmmu_irq() also gets the lock.

No, it is not disabled. But, ipmmu_irq() uses another mmu->lock. So, I think, there won't be a deadlock.

Or I really missed something?

If we worry about ipmmu_tlb_invalidate() which is called here (to perform a flush by request from P2M code, which manages a page table) and from the irq handler (to perform a flush to resume address translation), I could use a tasklet to schedule ipmmu_tlb_invalidate() from the irq handler then. This way we would get this serialized. What do you think?

Quoted 2 lines

# To be honest, in normal case, any irq on the current implementation

# should not happen though.

Agree here.

Quoted 8 lines
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8 -0
+    /*
+     * Destroy Root IPMMU domain which context is mapped to this Xen domain
+     * if exits.
+     */
+    if ( xen_domain->root_domain )
+        ipmmu_free_root_domain(xen_domain->root_domain);
+
+    spin_unlock(&xen_domain->lock);
Quoted 8 lines

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1169

Quoted 7 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7 -0
+
+    /*
+     * We assume that all master devices have already been detached from
+     * this Xen domain and there must be no associated Cache IPMMU domains
+     * in use.
+     */
+    ASSERT(list_empty(&xen_domain->cache_domains));

I think this should be in the spin lock held by &xen_domain->lock.

OK. Will put spin_unlock after it.

Quoted 13 lines
Quoted 13 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +12 -0
+    xfree(xen_domain);
+    dom_iommu(d)->arch.priv = NULL;
+}
+
+static const struct iommu_ops ipmmu_iommu_ops =
+{
+    .init            = ipmmu_iommu_domain_init,
+    .hwdom_init      = ipmmu_iommu_hwdom_init,
+    .teardown        = ipmmu_iommu_domain_teardown,
+    .iotlb_flush     = ipmmu_iotlb_flush,
+    .iotlb_flush_all = ipmmu_iotlb_flush_all,
+    .assign_device   = ipmmu_assign_device,
Quoted 8 lines

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1188

Quoted 7 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7 -0
+    .reassign_device = ipmmu_reassign_device,
+    .map_page        = arm_iommu_map_page,
+    .unmap_page      = arm_iommu_unmap_page,
+    .add_device      = ipmmu_add_device,
+};
+
+/* RCAR GEN3 product and cut information. */

"R-Car Gen3 SoCs" is better than "RCAR GEN3".

Will update.

Quoted 6 lines

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1191

Quoted 4 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +3 -0
+#define RCAR_PRODUCT_MASK    0x00007F00
+#define RCAR_PRODUCT_H3      0x00004F00
+#define RCAR_PRODUCT_M3      0x00005200

At least, I think we should be M3W, instead of M3. # FYI, M3-W and M3-W+ are the same value.

Will update.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#40 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

(+ Lars)

Hi,

Quoted 22 lines · Oleksandr
Quoted 22 lines
Quoted 22 lines

+ * you can found at:
+ *    url:
git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git
+ *    branch: v4.14.75-ltsi/rcar-3.9.6
+ *    commit: e206eb5b81a60e64c35fbc3a999b1a0db2b98044
+ * and Xen's SMMU driver:
+ *    xen/drivers/passthrough/arm/smmu.c
+ *
+ * Copyright (C) 2016-2019 EPAM Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; If not, see
<http://www.gnu.org/licenses/>.

Quoted 8 lines
Quoted 2 lines

I don't know that Xen license description rule, but since a few source files have SPDX-License-Identifier, can we also use it on the driver?

I am afraid, I don't know a correct answer for this question. I would leave this to maintainers.

I just followed sample copyright notice for GPL v2 License according to the document:

http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=CONTRIBUTING

The file CONTRIBUTING is only giving example of common example of license. So I think this is fine to use SPDX, the more they are already used. The only request is to put either SDPX or the full-blown text but not the two :). Lars, any objection?

I am quite in favor of SPDX because it is easier to find out the license. With the full-blown text, the text may slightly vary between licenses. For instance, the only difference between GPLv2 and GPLv2+ is ",or (at your option) any later version". I let you imagine how it can be easy to miss it when reviewing ;).

We had a discussion last year about using SPDX in Xen code base but I never got the time to formally suggest it.

Quoted 12 lines
Quoted 11 lines

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 42

Quoted 8 lines

+ */

xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6 -0
+
+#include <xen/delay.h>
+#include <xen/err.h>
+#include <xen/irq.h>
+#include <xen/lib.h>
+#include <xen/list.h>

I don't know that Xen passthrough driver rule though, doesn't here need #include <xen/iommu.h>? (The xen/sched.h seems to have it so that no compile error happens though.)

Quoted 1 line

Probably, yes, I should have included that header.

I am fine either way :). The indirect inclusion happens quite often and we only notice it when someone decide to rework the headers.

[...]

Quoted 14 lines
Quoted 12 lines

+/* Xen IOMMU ops */

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 811

Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8 -0
+static int __must_check ipmmu_iotlb_flush_all(struct domain *d)
+{
+    struct ipmmu_vmsa_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
+
+    if ( !xen_domain || !xen_domain->root_domain )
+        return 0;
+
+    spin_lock(&xen_domain->lock);

Is local irq is already disabled here?
If no, you should use spin_lock_irqsave() because the ipmmu_irq() also gets the lock.

Quoted 5 lines

No, it is not disabled. But, ipmmu_irq() uses another mmu->lock. So, I think, there won't be a deadlock.

Or I really missed something?

If we worry about ipmmu_tlb_invalidate() which is called here (to perform a flush by request from P2M code, which manages a page table) and from the irq handler (to perform a flush to resume address translation), I could use a tasklet to schedule ipmmu_tlb_invalidate() from the irq handler then. This way we would get this serialized. What do you think?

I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.

This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).

[...]

Quoted 20 lines
Quoted 11 lines
Quoted 11 lines

+    /*
+     * Destroy Root IPMMU domain which context is mapped to this Xen domain
+     * if exits.
+     */
+    if ( xen_domain->root_domain )
+        ipmmu_free_root_domain(xen_domain->root_domain);

xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +4 -0
+
+    spin_unlock(&xen_domain->lock);
+
+    /*
Quoted 8 lines
Quoted 7 lines

+     * We assume that all master devices have already been detached from
+     * this Xen domain and there must be no associated Cache IPMMU domains
+     * in use.
+     */
+    ASSERT(list_empty(&xen_domain->cache_domains));

I think this should be in the spin lock held by &xen_domain->lock.

Quoted 1 line

OK. Will put spin_unlock after it.

The spin_lock is actually pointless here. This is done when the domain is destroyed, so nobody should touch it.

If you think concurrent access can still happen, then you are going to be in deep trouble as you free the xen_domain (and therefore the spinlock) below :).

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#41 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Hi, Julien.
Sorry for the possible format issues.

Quoted 15 lines
Quoted 11 lines

No, it is not disabled. But, ipmmu_irq() uses another mmu->lock. So, I think, there won't be a deadlock.

Or I really missed something?

If we worry about ipmmu_tlb_invalidate() which is called here (to perform a flush by request from P2M code, which manages a page table) and from the irq handler (to perform a flush to resume address translation), I could use a tasklet to schedule ipmmu_tlb_invalidate() from the irq handler then. This way we would get this serialized. What do you think?

I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.

Quoted 9 lines

This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).

I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:

https://github.com/otyshchenko1/xen/blob/ipmmu_upstream2/xen/drivers/passthrough/arm/ipmmu-vmsa.c#L598

Sorry if I was unclear.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#42 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Quoted 1 line · Oleksandr Tyshchenko

Hi, Julien.

Hi,

Quoted 27 lines

Sorry for the possible format issues.

> No, it is not disabled. But, ipmmu_irq() uses another mmu->lock. So, I > think, there won't be a deadlock.
>
> Or I really missed something?
>
> If we worry about ipmmu_tlb_invalidate() which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to schedule ipmmu_tlb_invalidate() > from the irq handler then. This way we would get this serialized. What > do you think?

I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.

This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).

Quoted 5 lines

I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:

https://github.com/otyshchenko1/xen/blob/ipmmu_upstream2/xen/drivers/passthrough/arm/ipmmu-vmsa.c#L598

Sorry if I was unclear.

My mistake, I misread what you wrote.

I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?

Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#43 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Quoted 3 lines

Hi,

Hi, Julien.

Quoted 33 lines
Quoted 33 lines

Sorry for the possible format issues.

     > No, it is not disabled. But, ipmmu_irq() uses another mmu->lock. So, I > think, there won't be a deadlock.
     >
     > Or I really missed something?
     >
     > If we worry about ipmmu_tlb_invalidate() which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to schedule ipmmu_tlb_invalidate() > from the irq handler then. This way we would get this serialized. What > do you think?

    I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.

    This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).

I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:

https://github.com/otyshchenko1/xen/blob/ipmmu_upstream2/xen/drivers/passthrough/arm/ipmmu-vmsa.c#L598

Quoted 7 lines
Quoted 1 line

Sorry if I was unclear.

My mistake, I misread what you wrote.

I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?

Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?

Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.

So, my worry about calling ipmmu_tlb_invalidate() directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.

But, with one remark, as you have already noted, we are not trying to handle/fix this fault (update page table entries), driver doesn't manage page table and is not aware what the page table is. What is more, it is unclear what actually need to be fixed in the page table which is a CPU page table as the same time.

I have heard there is a break-before-make sequence when updating the page table. So, if device in a domain is issuing DMA somewhere in the middle of updating a page table, theoretically, we might hit into this fault. In this case the page table is correct and we don't need to fix anything...   Being honest, I have never seen a fault caused by break-before-make sequence.

Quoted 3 lines

Cheers,

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#44 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Hi,

Removing Lars there is no need to spam him with technical discussion :)

Quoted 49 lines · Oleksandr
Quoted 3 lines

Hi,

Hi, Julien.

Quoted 33 lines
Quoted 33 lines

Sorry for the possible format issues.

     > No, it is not disabled. But, ipmmu_irq() uses another mmu->lock. So, I > think, there won't be a deadlock.
     >
     > Or I really missed something?
     >
     > If we worry about ipmmu_tlb_invalidate() which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to schedule ipmmu_tlb_invalidate() > from the irq handler then. This way we would get this serialized. What > do you think?

    I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.

    This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).

I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:

https://github.com/otyshchenko1/xen/blob/ipmmu_upstream2/xen/drivers/passthrough/arm/ipmmu-vmsa.c#L598

Quoted 7 lines
Quoted 1 line

Sorry if I was unclear.

My mistake, I misread what you wrote.

I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?

Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?

Quoted 6 lines

Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.

So, my worry about calling ipmmu_tlb_invalidate() directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.

Well, I don't have the TRM... so my point of reference is Linux. Why does upstream not do the TLB flush?

I have been told this is an errata on the IPMMU. Is it related to the series posted on linux-iommu [1]?

Quoted 4 lines

But, with one remark, as you have already noted, we are not trying to handle/fix this fault (update page table entries), driver doesn't manage page table and is not aware what the page table is. What is more, it is unclear what actually need to be fixed in the page table which is a CPU page table as the same time.

I have heard there is a break-before-make sequence when updating the page table. So, if device in a domain is issuing DMA somewhere in the middle of updating a page table, theoretically, we might hit into this fault. In this case the page table is correct and we don't need to fix anything...   Being honest, I have never seen a fault caused by break-before-make sequence.

Ok, so it looks like you are trying to fix [1]. My first concern here is there are no ground for someone without access to the TRM why this is done.

Furthermore, AFAICT, the patch series never reached upstream. So is it present on all revision of GEN3?

Cheers,

[1] https://lore.kernel.org/linux-iommu/1485348842-23712-1-git-send-email-yoshihiro.shimoda.uh@renesas.com/

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#45 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Hi

Quoted 42 lines
Quoted 42 lines
Quoted 40 lines
Quoted 28 lines

Sorry for the possible format issues.

     > No, it is not disabled. But, ipmmu_irq() uses another mmu->lock. So, I > think, there won't be a deadlock.
     >
     > Or I really missed something?
     >
     > If we worry about ipmmu_tlb_invalidate() which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to schedule ipmmu_tlb_invalidate() > from the irq handler then. This way we would get this serialized. What > do you think?

    I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.

    This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).

Quoted 6 lines

I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:

https://github.com/otyshchenko1/xen/blob/ipmmu_upstream2/xen/drivers/passthrough/arm/ipmmu-vmsa.c#L598

Sorry if I was unclear.

My mistake, I misread what you wrote.

I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?

Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?

Quoted 8 lines
Quoted 6 lines

Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.

So, my worry about calling ipmmu_tlb_invalidate() directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.

Well, I don't have the TRM... so my point of reference is Linux. Why does upstream not do the TLB flush?

I have no idea regarding that.

Quoted 3 lines

I have been told this is an errata on the IPMMU. Is it related to the series posted on linux-iommu [1]?

I don't think, the TLB flush we are speaking about, is related to that series [1] somehow. This TLB flush, I think, is just the last step in a sequence of actions which should be performed when the error occurs, no more no less. This is how I understand this.

Quoted 7 lines
Quoted 5 lines

But, with one remark, as you have already noted, we are not trying to handle/fix this fault (update page table entries), driver doesn't manage page table and is not aware what the page table is. What is more, it is unclear what actually need to be fixed in the page table which is a CPU page table as the same time.

I have heard there is a break-before-make sequence when updating the page table. So, if device in a domain is issuing DMA somewhere in the middle of updating a page table, theoretically, we might hit into this fault. In this case the page table is correct and we don't need to fix anything...   Being honest, I have never seen a fault caused by break-before-make sequence.

Ok, so it looks like you are trying to fix [1]. My first concern here is there are no ground for someone without access to the TRM why this is done.

No, I am definitely not trying to fix [1]. I just follow the BSP driver I am based on, which in turn follows the TRM. I can extend a comment in the code before calling ipmmu_tlb_invalidate().

Quoted 2 lines

Furthermore, AFAICT, the patch series never reached upstream. So is it present on all revision of GEN3?

I think, that the newest SoCs revisions (ES 3.0) this driver is supposed to support only, are *not* affected by that errata. And *not* require such workaround.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#46 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Quoted 54 lines · Oleksandr

Hi

Quoted 42 lines
Quoted 42 lines
Quoted 40 lines
Quoted 28 lines

Sorry for the possible format issues.

     > No, it is not disabled. But, ipmmu_irq() uses another mmu->lock. So, I > think, there won't be a deadlock.
     >
     > Or I really missed something?
     >
     > If we worry about ipmmu_tlb_invalidate() which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to schedule ipmmu_tlb_invalidate() > from the irq handler then. This way we would get this serialized. What > do you think?

    I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.

    This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).

Quoted 6 lines

I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:

https://github.com/otyshchenko1/xen/blob/ipmmu_upstream2/xen/drivers/passthrough/arm/ipmmu-vmsa.c#L598

Sorry if I was unclear.

My mistake, I misread what you wrote.

I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?

Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?

Quoted 8 lines
Quoted 6 lines

Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.

So, my worry about calling ipmmu_tlb_invalidate() directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.

Well, I don't have the TRM... so my point of reference is Linux. Why does upstream not do the TLB flush?

Quoted 7 lines

I have no idea regarding that. >

Quoted 3 lines

I have been told this is an errata on the IPMMU. Is it related to the series posted on linux-iommu [1]?

I don't think, the TLB flush we are speaking about, is related to that series [1] somehow. This TLB flush, I think, is just the last step in a sequence of actions which should be performed when the error occurs, no more no less. This is how I understand this.

If you have to flush the TLBs in the IRQ context then something has gone really wrong.

I don't deny that Break-Before-Make is an issue. However, if it is handled correctly in the P2M code. You should only be there because there are no mapping in the TLBs for the address accessed. So flushing the TLBs should be unnecessary, unless your TLB is also caching invalid entry?

Quoted 9 lines
Quoted 7 lines
Quoted 5 lines

But, with one remark, as you have already noted, we are not trying to handle/fix this fault (update page table entries), driver doesn't manage page table and is not aware what the page table is. What is more, it is unclear what actually need to be fixed in the page table which is a CPU page table as the same time.

I have heard there is a break-before-make sequence when updating the page table. So, if device in a domain is issuing DMA somewhere in the middle of updating a page table, theoretically, we might hit into this fault. In this case the page table is correct and we don't need to fix anything...   Being honest, I have never seen a fault caused by break-before-make sequence.

Ok, so it looks like you are trying to fix [1]. My first concern here is there are no ground for someone without access to the TRM why this is done.

No, I am definitely not trying to fix [1]. I just follow the BSP driver I am based on, which in turn follows the TRM. I can extend a comment in the code before calling ipmmu_tlb_invalidate().

The fact that the code is in the BSP and not in Linux is worrying me. The commit message in the BSP is quite unhelpful to determine the exact reason.

It either means Linux rejected the patch or this was not submitted. Either way, this should be understood why such discrepancy.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#47 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Hi, Julien.

Quoted 55 lines
Quoted 55 lines
Quoted 42 lines
Quoted 42 lines
Quoted 40 lines
Quoted 28 lines

Sorry for the possible format issues.

     > No, it is not disabled. But, ipmmu_irq() uses another mmu->lock. So, I > think, there won't be a deadlock.
     >
     > Or I really missed something?
     >
     > If we worry about ipmmu_tlb_invalidate() which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to schedule ipmmu_tlb_invalidate() > from the irq handler then. This way we would get this serialized. What > do you think?

    I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.

    This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).

Quoted 6 lines

I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:

https://github.com/otyshchenko1/xen/blob/ipmmu_upstream2/xen/drivers/passthrough/arm/ipmmu-vmsa.c#L598

Sorry if I was unclear.

My mistake, I misread what you wrote.

I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?

Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?

Quoted 8 lines
Quoted 6 lines

Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.

So, my worry about calling ipmmu_tlb_invalidate() directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.

Well, I don't have the TRM... so my point of reference is Linux. Why does upstream not do the TLB flush?

I have no idea regarding that. >

Quoted 2 lines
Quoted 7 lines
Quoted 3 lines
Quoted 1 line

I have been told this is an errata on the IPMMU. Is it related to the series posted on linux-iommu [1]?

I don't think, the TLB flush we are speaking about, is related to that series [1] somehow. This TLB flush, I think, is just the last step in a sequence of actions which should be performed when the error occurs, no more no less. This is how I understand this.

If you have to flush the TLBs in the IRQ context then something has gone really wrong.

I don't deny that Break-Before-Make is an issue. However, if it is handled correctly in the P2M code. You should only be there because there are no mapping in the TLBs for the address accessed. So flushing the TLBs should be unnecessary, unless your TLB is also caching invalid entry?

Sorry, I don't quite understand why we need to worry about this flush too much for a case which won't occur in normal condition (if everything is correct). Why we can't just consider this flush as a required action, which needed to exit from the error state and resume stopped address translation request. The same required action as "clearing error status flags" before. We are not trying to understand, why is it so necessary to clear error flags when error happens, or can we end up without clearing it, for example. We just follow what described in document. The same, I think, we have for that flush, if described, then should be followed. Looks like this flush acts as a trigger to unblock stopped transaction in that particular case.

Different H/W could have different restoring sequences. Some H/W requires just clearing error status, other H/W requires full re-initialization in a specific order to recover from the error state.

Please correct me if I am wrong.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

But, with one remark, as you have already noted, we are not trying to handle/fix this fault (update page table entries), driver doesn't manage page table and is not aware what the page table is. What is more, it is unclear what actually need to be fixed in the page table which is a CPU page table as the same time.

I have heard there is a break-before-make sequence when updating the page table. So, if device in a domain is issuing DMA somewhere in the middle of updating a page table, theoretically, we might hit into this fault. In this case the page table is correct and we don't need to fix anything...   Being honest, I have never seen a fault caused by break-before-make sequence.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Ok, so it looks like you are trying to fix [1]. My first concern here is there are no ground for someone without access to the TRM why this is done.

No, I am definitely not trying to fix [1]. I just follow the BSP driver I am based on, which in turn follows the TRM. I can extend a comment in the code before calling ipmmu_tlb_invalidate().

The fact that the code is in the BSP and not in Linux is worrying me. The commit message in the BSP is quite unhelpful to determine the exact reason.

It either means Linux rejected the patch or this was not submitted. Either way, this should be understood why such discrepancy.

I failed to find something similar in the ML. So, probably, was not submitted. Hope, we will be able to clarify a reason.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#48 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Hi Oleksandr,

Quoted 57 lines · Oleksandr
Quoted 57 lines
Quoted 57 lines
Quoted 42 lines
Quoted 42 lines
Quoted 40 lines
Quoted 28 lines

Sorry for the possible format issues.

     > No, it is not disabled. But, ipmmu_irq() uses another mmu->lock. So, I > think, there won't be a deadlock.
     >
     > Or I really missed something?
     >
     > If we worry about ipmmu_tlb_invalidate() which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to schedule ipmmu_tlb_invalidate() > from the irq handler then. This way we would get this serialized. What > do you think?

    I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.

    This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).

Quoted 6 lines

I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:

https://github.com/otyshchenko1/xen/blob/ipmmu_upstream2/xen/drivers/passthrough/arm/ipmmu-vmsa.c#L598

Sorry if I was unclear.

My mistake, I misread what you wrote.

I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?

Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?

Quoted 8 lines
Quoted 6 lines

Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.

So, my worry about calling ipmmu_tlb_invalidate() directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.

Well, I don't have the TRM... so my point of reference is Linux. Why does upstream not do the TLB flush?

I have no idea regarding that. >

Quoted 3 lines

I have been told this is an errata on the IPMMU. Is it related to the series posted on linux-iommu [1]?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

I don't think, the TLB flush we are speaking about, is related to that series [1] somehow. This TLB flush, I think, is just the last step in a sequence of actions which should be performed when the error occurs, no more no less. This is how I understand this.

If you have to flush the TLBs in the IRQ context then something has gone really wrong.

I don't deny that Break-Before-Make is an issue. However, if it is handled correctly in the P2M code. You should only be there because there are no mapping in the TLBs for the address accessed. So flushing the TLBs should be unnecessary, unless your TLB is also caching invalid entry?

Sorry, I don't quite understand why we need to worry about this flush too much for a case which won't occur in normal condition (if everything is correct). Why we can't just consider this flush as a required action,

A translation error can be easy to reach. For instance if the guest does not program the Device correctly and request to access an address that is not mapped.

Quoted 1 line

which needed to exit from the error state and resume stopped address translation request. The same required action as "clearing error status flags" before. We are not trying to understand, why is it so necessary to clear error flags when error happens, or can we end up without clearing it, for example. We just follow what described in document. The same, I think, we have for that flush, if described, then should be followed. Looks like this flush acts as a trigger to unblock stopped transaction in that particular case.

What will actually happen if the transaction fail again? For instance, if the IOVA was not mapped. Will you receive the interrupt again? If so, are you going to make the flush again and again until the guest is killed?

Quoted 4 lines

Different H/W could have different restoring sequences. Some H/W requires just clearing error status, other H/W requires full re-initialization in a specific order to recover from the error state.

Please correct me if I am wrong.

I am not confident to accept any code that I don't understand or I don't find sensible. As I pointed out in my previous e-mail, this hasn't reached upstream so something looks quite fishy here.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#49 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Hi Julien.

Sorry for the possible format issues.

чт, 8 авг. 2019 г., 23:32 Julien Grall <[email protected]>:

Quoted 105 lines

Hi Oleksandr,

Quoted 41 lines · Oleksandr
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines

Sorry for the possible format issues.

> No, it is not disabled. But, ipmmu_irq() uses another
mmu->lock. So, I
> think, there won't be a deadlock.
>
> Or I really missed something?
>
> If we worry about ipmmu_tlb_invalidate() which is called
here (to
> perform a flush by request from P2M code, which manages a
page table)
> and from the irq handler (to perform a flush to resume
address
> translation), I could use a tasklet to schedule
ipmmu_tlb_invalidate()
> from the irq handler then. This way we would get this
serialized. What
> do you think?

I am afraid a tasklet is not an option. You need to perform
the TLB
flush when requested otherwise you are introducing a security
issue.

This is because as soon as a region is unmapped in the page
table, we
remove the drop the reference on any page backing that
region. When the
reference is dropped to zero, the page can be reallocated to
another
domain or even Xen. If the TLB flush happen after, then the
guest may
still be able to access the page for a short time if the
translation has
been cached by the any TLB (IOMMU, Processor).

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:

https://github.com/otyshchenko1/xen/blob/ipmmu_upstream2/xen/drivers/passthrough/arm/ipmmu-vmsa.c#L598

Quoted 52 lines
Quoted 51 lines
Quoted 35 lines
Quoted 28 lines
Quoted 18 lines
Quoted 6 lines
Quoted 3 lines

Sorry if I was unclear.

My mistake, I misread what you wrote.

Quoted 7 lines

I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?

Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?

Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.

Quoted 9 lines

So, my worry about calling ipmmu_tlb_invalidate() directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.

Quoted 2 lines

Well, I don't have the TRM... so my point of reference is Linux. Why does upstream not do the TLB flush?

I have no idea regarding that. >

Quoted 2 lines
Quoted 7 lines
Quoted 2 lines

I have been told this is an errata on the IPMMU. Is it related to the series posted on linux-iommu [1]?

I don't think, the TLB flush we are speaking about, is related to that series [1] somehow. This TLB flush, I think, is just the last step in a sequence of actions which should be performed when the error occurs, no more no less. This is how I understand this.

If you have to flush the TLBs in the IRQ context then something has gone really wrong.

I don't deny that Break-Before-Make is an issue. However, if it is handled correctly in the P2M code. You should only be there because there are no mapping in the TLBs for the address accessed. So flushing the TLBs should be unnecessary, unless your TLB is also caching invalid entry?

Quoted 8 lines
Quoted 3 lines

Sorry, I don't quite understand why we need to worry about this flush too much for a case which won't occur in normal condition (if everything is correct). Why we can't just consider this flush as a required action,

A translation error can be easy to reach. For instance if the guest does not program the Device correctly and request to access an address that is not mapped.

Yes, I understand these bits. But, I wrote that error wouldn't occur in normal condition (if everything was correct).

Quoted 11 lines
Quoted 10 lines

which needed to exit from the error state and resume stopped address translation request. The same required action as "clearing error status flags" before. We are not trying to understand, why is it so necessary to clear error flags when error happens, or can we end up without clearing it, for example. We just follow what described in document. The same, I think, we have for that flush, if described, then should be followed. Looks like this flush acts as a trigger to unblock stopped transaction in that particular case.

Quoted 5 lines

What will actually happen if the transaction fail again? For instance, if the IOVA was not mapped. Will you receive the interrupt again? If so, are you going to make the flush again and again until the guest is killed?

This is a good question. I think, if address is not mapped, the transaction will fail again and we will get the interrupt again. Not sure, until the guest is killed or until the driver in the guest detects timeout and cancels DMA. Let's consider the worst case, until the guest is killed.

So my questions are what do you think would be the proper driver's behavior in that case? Do nothing and don't even try to resolve error condition/unblock translation at the first page fault, or give it a few attempts, or unblock every time. How does the SMMU driver act in such situation?

Quite clear, if we get a fault, then address is not mapped. I think, it can be both: by issuing wrong address (baggy driver, malicious driver) or by race (unlikely). If this is the real race (device hits brake-before-make, for example), we could give it another attempt, for example. Looks like we need some mechanism to deploy faulted address to P2M code (which manages page table) to analyze? Or it is not worth doing that?

Quoted 5 lines
Quoted 5 lines

Different H/W could have different restoring sequences. Some H/W requires just clearing error status, other H/W requires full re-initialization in a specific order to recover from the error state.

Quoted 7 lines
Quoted 1 line

Please correct me if I am wrong.

I am not confident to accept any code that I don't understand or I don't find sensible. As I pointed out in my previous e-mail, this hasn't reached upstream so something looks quite fishy here.

As I answered in previous e-mail, I hope, we will be able to clarify a reason why this hasn't reached upstream.

Quoted 1 line

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#50 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

(+ Robin)

Quoted 1 line · Oleksandr Tyshchenko

Hi Julien.

Hi,

Quoted 115 lines

Sorry for the possible format issues.

чт, 8 авг. 2019 г., 23:32 Julien Grall <[email protected] <mailto:[email protected]>>:

Hi Oleksandr,

On 8/8/19 8:29 PM, Oleksandr wrote:
>>>>>>
>>>>>>> Sorry for the possible format issues.
>>>>>>>
>>>>>>>
>>>>>>>      > No, it is not disabled. But, ipmmu_irq() uses another >>>>>>> mmu->lock. So, I
>>>>>>>      > think, there won't be a deadlock.
>>>>>>>      >
>>>>>>>      > Or I really missed something?
>>>>>>>      >
>>>>>>>      > If we worry about ipmmu_tlb_invalidate() which is called >>>>>>> here (to
>>>>>>>      > perform a flush by request from P2M code, which manages a >>>>>>> page table)
>>>>>>>      > and from the irq handler (to perform a flush to resume >>>>>>> address
>>>>>>>      > translation), I could use a tasklet to schedule >>>>>>> ipmmu_tlb_invalidate()
>>>>>>>      > from the irq handler then. This way we would get this >>>>>>> serialized. What
>>>>>>>      > do you think?
>>>>>>>
>>>>>>>     I am afraid a tasklet is not an option. You need to perform >>>>>>> the TLB
>>>>>>>     flush when requested otherwise you are introducing a security >>>>>>> issue.
>>>>>>>
>>>>>>>     This is because as soon as a region is unmapped in the page >>>>>>> table, we
>>>>>>>     remove the drop the reference on any page backing that >>>>>>> region. When the
>>>>>>>     reference is dropped to zero, the page can be reallocated to >>>>>>> another
>>>>>>>     domain or even Xen. If the TLB flush happen after, then the >>>>>>> guest may
>>>>>>>     still be able to access the page for a short time if the >>>>>>> translation has
>>>>>>>     been cached by the any TLB (IOMMU, Processor). >>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I understand this. I am not proposing to delay a requested by P2M >>>>>>> code TLB flush in any case. I just propose to issue TLB flush >>>>>>> (which we have to perform in case of page faults, to resolve >>>>>>> error condition and resume translations) from a tasklet rather >>>>>>> than from interrupt handler directly. This is the TLB flush I am >>>>>>> speaking about:
>>>>>>>
>>>>>>>
https://github.com/otyshchenko1/xen/blob/ipmmu_upstream2/xen/drivers/passthrough/arm/ipmmu-vmsa.c#L598

>>>>>>>
>>>>>>>
>>>>>>> Sorry if I was unclear.
>>>>>>
>>>>>> My mistake, I misread what you wrote.
>>>>>>
>>>>>> I found the flush in the renesas-bsp and not Linux upstream but it >>>>>> is not clear why this is actually required. You are not fixing any >>>>>> translation error. So what this flush will do?
>>>>>>
>>>>>> Regarding the placement of the flush, then if you execute in a >>>>>> tasklet it will likely be done later on when the IRQ has been >>>>>> acknowledge. What's the implication to delay it?
>>>>>
>>>>>
>>>>> Looks like, there is no need to put this flush into a tasklet. As I >>>>> understand from Shimoda-san's answer it is OK to call flush here. >>>>>
>>>>> So, my worry about calling ipmmu_tlb_invalidate() directly from the >>>>> interrupt handler is not actual anymore.
>>>>> ----------
>>>>> This is my understanding regarding the flush purpose here. This >>>>> code, just follows the TRM, no more no less,
>>>>> which mentions about a need to flush TLB after clearing error >>>>> status register and updating a page table entries (which, I assume, >>>>> means to resolve a reason why translation/page fault error actually >>>>> have happened) to resume address translation request. >>>>
>>>> Well, I don't have the TRM... so my point of reference is Linux. Why >>>> does upstream not do the TLB flush?
>>>
>>> I have no idea regarding that. >
>>>
>>>>
>>>>
>>>> I have been told this is an errata on the IPMMU. Is it related to >>>> the series posted on linux-iommu [1]?
>>>
>>> I don't think, the TLB flush we are speaking about, is related to >>> that series [1] somehow. This TLB flush, I think, is just the last >>> step in a sequence of actions which should be performed when the >>> error occurs, no more no less. This is how I understand this. >>
>> If you have to flush the TLBs in the IRQ context then something has >> gone really wrong.
>>
>> I don't deny that Break-Before-Make is an issue. However, if it is >> handled correctly in the P2M code. You should only be there because >> there are no mapping in the TLBs for the address accessed. So flushing >> the TLBs should be unnecessary, unless your TLB is also caching >> invalid entry?
>
> Sorry, I don't quite understand why we need to worry about this flush > too much for a case which won't occur in normal condition (if everything > is correct). Why we can't just consider this flush as a required action,

Quoted 6 lines

A translation error can be easy to reach. For instance if the guest does not program the Device correctly and request to access an address that is not mapped.

Yes, I understand these bits. But, I wrote that error wouldn't occur in normal condition (if everything was correct).

I don't understand your point here. Whether this is in an error path or correct path, we should be able to understand the reason behind it. Otherwise, error path would become the wild west...

Quoted 20 lines

> which needed to exit from the error state and resume stopped address > translation request. The same required action as "clearing error status > flags" before. We are not trying to understand, why is it so necessary > to clear error flags when error happens, or can we end up without > clearing it, for example. We just follow what described in document. The > same, I think, we have for that flush, if described, then should be > followed. Looks like this flush acts as a trigger to unblock stopped > transaction in that particular case.

What will actually happen if the transaction fail again? For instance, if the IOVA was not mapped. Will you receive the interrupt again? If so, are you going to make the flush again and again until the guest is killed?

Quoted 3 lines

This is a good question. I think, if address is not mapped, the transaction will fail again and we will get the interrupt again. Not sure, until the guest is killed or until the driver in the guest detects timeout and cancels DMA. Let's consider the worst case, until the guest is killed.

So my questions are what do you think would be the proper driver's behavior in that case? Do nothing and don't even try to resolve error condition/unblock translation at the first page fault, or give it a few attempts, or unblock every time.

I will answer back with a question here. How is the TLB flush is going to unblock anything? The more you are not fixing any error condition here... And the print "Unhandled fault" just afterwards clearly leads to think that there are very little chance the fault has been resolved.

Quoted 1 line

How does the SMMU driver act in such situation?

I have CCed Robin who knows better than me the SMMU driver. Though it is the Linux one but Xen is based on it.

From my understanding, it is implementation defined whether the SMMU supports stalling a transaction on fault. AFAICT, the current Xen driver will just terminate the transaction and therefore the client transaction behave as RAZ/WI.

Quoted 2 lines

Quite clear, if we get a fault, then address is not mapped. I think, it can be both: by issuing wrong address (baggy driver, malicious driver) or by race (unlikely). If this is the real race (device hits brake-before-make, for example), we could give it another attempt, for example. Looks like we need some mechanism to deploy faulted address to P2M code (which manages page table) to analyze? Or it is not worth doing that?

You seem to speak about break-before-make as it was an error. Break-Before-Make is just a sequence to prevent the TLB walker to cache both old and new mapping at the same time. At a given point the IOVA translation can only be:
1) The old physical address
2) No address -> result to a fault
3) The new physical address

1) and 3) should not result to a fault. 2) will result to a fault but then the TLB should not cache invalid entry, right?

In order to see 2), we always flush the TLBs after removing the old physical address.

Unfortunately, some of the IOMMUs are not able to restart transactions, Xen currently avoids to flush the TLBs after 2). So you may be able to see both mapping at the same time.

Looking at your driver, I believe you would have the flag IMSTR.MHIT (multiple tlb hits) set because this is the condition we are trying to prevent with break-before-make. The comment in the code leads to think this is a fault error, so I am not sure why you would recover here...

If your IOMMU is able to stall transaction, then it would be best if we properly handle break-before-make with it.

Overall, it feels to me the TLB flush is here for a different reason.

Quoted 9 lines

>
> Different H/W could have different restoring sequences. Some H/W > requires just clearing error status, other H/W requires full > re-initialization in a specific order to recover from the error state. >
> Please correct me if I am wrong.

Quoted 6 lines

I am not confident to accept any code that I don't understand or I don't find sensible. As I pointed out in my previous e-mail, this hasn't reached upstream so something looks quite fishy here.

As I answered in previous e-mail, I hope, we will be able to clarify a reason why this hasn't reached upstream.

Thank you.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#51 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Hi, Julien

Quoted 7 lines
Quoted 7 lines

    What will actually happen if the transaction fail again? For instance, if the IOVA was not mapped. Will you receive the interrupt again? If so, are you going to make the flush again and again until the guest is killed?

Quoted 5 lines
Quoted 3 lines

This is a good question. I think, if address is not mapped, the transaction will fail again and we will get the interrupt again. Not sure, until the guest is killed or until the driver in the guest detects timeout and cancels DMA. Let's consider the worst case, until the guest is killed.

So my questions are what do you think would be the proper driver's behavior in that case? Do nothing and don't even try to resolve error condition/unblock translation at the first page fault, or give it a few attempts, or unblock every time.

I will answer back with a question here. How is the TLB flush is going to unblock anything? The more you are not fixing any error condition here... And the print "Unhandled fault" just afterwards clearly leads to think that there are very little chance the fault has been resolved.

Now I understand your point. This really makes sense.

Quoted 6 lines
Quoted 2 lines

How does the SMMU driver act in such situation?

I have CCed Robin who knows better than me the SMMU driver. Though it is the Linux one but Xen is based on it.

From my understanding, it is implementation defined whether the SMMU supports stalling a transaction on fault. AFAICT, the current Xen driver will just terminate the transaction and therefore the client transaction behave as RAZ/WI.

I got it. So, sounds like the client won't be able to do something bad, and we won't receive an interrupt storm here in Xen.

Quoted 11 lines
Quoted 4 lines

Quite clear, if we get a fault, then address is not mapped. I think, it can be both: by issuing wrong address (baggy driver, malicious driver) or by race (unlikely). If this is the real race (device hits brake-before-make, for example), we could give it another attempt, for example. Looks like we need some mechanism to deploy faulted address to P2M code (which manages page table) to analyze? Or it is not worth doing that?

You seem to speak about break-before-make as it was an error. Break-Before-Make is just a sequence to prevent the TLB walker to cache both old and new mapping at the same time. At a given point the IOVA translation can only be:
   1) The old physical address
   2) No address -> result to a fault
   3) The new physical address

1) and 3) should not result to a fault. 2) will result to a fault but then the TLB should not cache invalid entry, right?

right.

Quoted 8 lines

In order to see 2), we always flush the TLBs after removing the old physical address.

Unfortunately, some of the IOMMUs are not able to restart transactions, Xen currently avoids to flush the TLBs after 2). So you may be able to see both mapping at the same time.

Looking at your driver, I believe you would have the flag IMSTR.MHIT (multiple tlb hits) set because this is the condition we are trying to prevent with break-before-make. The comment in the code leads to think this is a fault error, so I am not sure why you would recover here...

If your IOMMU is able to stall transaction, then it would be best if we properly handle break-before-make with it.

Thank you for the detailed answer. I would like to say that I have never seen Multiple tlb hits error raised by IPMMU in Xen.

Quoted 2 lines

Overall, it feels to me the TLB flush is here for a different reason.

I will drop this TLB flush from interrupt handler until clarified.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#52 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

In reply to Julien Grall

Hi, Julien.

Quoted 11 lines
Quoted 5 lines

I am afraid, I don't know a correct answer for this question. I would leave this to maintainers.

I just followed sample copyright notice for GPL v2 License according to the document:

http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=CONTRIBUTING

The file CONTRIBUTING is only giving example of common example of license. So I think this is fine to use SPDX, the more they are already used. The only request is to put either SDPX or the full-blown text but not the two :). Lars, any objection?

I am quite in favor of SPDX because it is easier to find out the license. With the full-blown text, the text may slightly vary between licenses. For instance, the only difference between GPLv2 and GPLv2+ is ",or (at your option) any later version". I let you imagine how it can be easy to miss it when reviewing ;).

We had a discussion last year about using SPDX in Xen code base but I never got the time to formally suggest it.

I tried to locate files in Xen where SPDX is used. After finding only nospec.h I got an incorrect opinion this is not popular in Xen))

Just to clarify. So the title for the driver should be the following (if there are no objections):

// SPDX-License-Identifier: GPL-2.0
/*
 * xen/drivers/passthrough/arm/ipmmu-vmsa.c
 *
 * Driver for the Renesas IPMMU-VMSA found in R-Car Gen3 SoCs.
 *
 * Copyright (C) 2014-2019 Renesas Electronics Corporation
 *
 * The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU)
 * which provides address translation and access protection functionalities
 * to processing units and interconnect networks.
 *
 * Please note, current driver is supposed to work only with newest R-Car Gen3
 * SoCs revisions which IPMMU hardware supports stage 2 translation table format
 * and is able to use CPU's P2M table as is.
 *
 * Based on Linux's IPMMU-VMSA driver from Renesas BSP:
 *    drivers/iommu/ipmmu-vmsa.c
 * you can found at:
 *    url: git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git
 *    branch: v4.14.75-ltsi/rcar-3.9.6
 *    commit: e206eb5b81a60e64c35fbc3a999b1a0db2b98044
 * and Xen's SMMU driver:
 *    xen/drivers/passthrough/arm/smmu.c
 *
 * Copyright (C) 2016-2019 EPAM Systems Inc.
 */

Answer to myself:

Looks like, the same I have to do with all newly added files in this series (iommu_fwspec, etc).

Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 12 lines

+    /*
+     * Destroy Root IPMMU domain which context is mapped to this Xen domain
+     * if exits.
+     */
+    if ( xen_domain->root_domain )
+        ipmmu_free_root_domain(xen_domain->root_domain);

xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5 -0
+
+    spin_unlock(&xen_domain->lock);
+
+    /*
+     * We assume that all master devices have already been 
Quoted 6 lines

detached from
+     * this Xen domain and there must be no associated Cache IPMMU domains
+     * in use.
+     */
+    ASSERT(list_empty(&xen_domain->cache_domains));

I think this should be in the spin lock held by &xen_domain->lock.

Quoted 5 lines
Quoted 1 line

OK. Will put spin_unlock after it.

The spin_lock is actually pointless here. This is done when the domain is destroyed, so nobody should touch it.

If you think concurrent access can still happen, then you are going to be in deep trouble as you free the xen_domain (and therefore the spinlock) below :).

Indeed, this is pointless. We don't really expect any other operations with the domain which is being destroyed. No assign/deassign devices, no flush, no map, nothing...

Quoted 2 lines
Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#53 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

In reply to Julien Grall

On 07/08/2019, 20:15, "Julien Grall" <[email protected]> wrote:

(+ Lars)

Hi,

On 8/7/19 5:01 PM, Oleksandr wrote:
>>> + * you can found at:
>>> + * url:
>>> git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git
>>> + * branch: v4.14.75-ltsi/rcar-3.9.6
>>> + * commit: e206eb5b81a60e64c35fbc3a999b1a0db2b98044
>>> + * and Xen's SMMU driver:
>>> + * xen/drivers/passthrough/arm/smmu.c
>>> + *
>>> + * Copyright (C) 2016-2019 EPAM Systems Inc.
>>> + *
>>> + * This program is free software; you can redistribute it and/or >>> + * modify it under the terms and conditions of the GNU General Public >>> + * License, version 2, as published by the Free Software Foundation. >>> + *
>>> + * This program is distributed in the hope that it will be useful, >>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU >>> + * General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public >>> + * License along with this program; If not, see
>>> <http://www.gnu.org/licenses/>.
>> I don't know that Xen license description rule, but since a few source >> files have
>> SPDX-License-Identifier, can we also use it on the driver? >
> I am afraid, I don't know a correct answer for this question. I would > leave this to maintainers.
>
> I just followed sample copyright notice for GPL v2 License according to > the document:
>
> http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=CONTRIBUTING

The file CONTRIBUTING is only giving example of common example of license. So I think this is fine to use SPDX, the more they are already used. The only request is to put either SDPX or the full-blown text but not the two :). Lars, any objection?

I am quite in favor of SPDX because it is easier to find out the license. With the full-blown text, the text may slightly vary between licenses. For instance, the only difference between GPLv2 and GPLv2+ is ",or (at your option) any later version". I let you imagine how it can be easy to miss it when reviewing ;).

We had a discussion last year about using SPDX in Xen code base but I never got the time to formally suggest it.

I did not push it either.

In the past one of the committers had major objections against SPDX, but after a conversation last year and changes to the latest version of SPDX he dropped these.

The only remaining objection was to have both SPDX identifier AND a license in the same file. The argument against it is: what does it mean if they contradict each other? To be fair that is a valid concern.

I am not sure it is a good idea to introduce SPDX piecemeal. It would be much better to
a) agree it
b) transform the codebase using a tool
rather than introducing it piecemeal

Lars

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#54 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

In reply to Oleksandr

Hi Oleksandr-san,

Quoted 6 lines

From: Oleksandr, Sent: Thursday, August 8, 2019 1:01 AM

Hi, Shimoda-san.

Thank you for the review.

You're welcome.

<snip>

Quoted 14 lines
Quoted 12 lines

+/* Xen IOMMU ops */

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 811

Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8 -0
+static int __must_check ipmmu_iotlb_flush_all(struct domain *d)
+{
+    struct ipmmu_vmsa_xen_domain *xen_domain = dom_iommu(d)->arch.priv;
+
+    if ( !xen_domain || !xen_domain->root_domain )
+        return 0;
+
+    spin_lock(&xen_domain->lock);

Is local irq is already disabled here?
If no, you should use spin_lock_irqsave() because the ipmmu_irq() also gets the lock.

Quoted 4 lines

No, it is not disabled. But, ipmmu_irq() uses another mmu->lock. So, I think, there won't be a deadlock.

Or I really missed something?

You're correct. I didn't realized that ipmmu_irq() used another mmu->lock.

Quoted 6 lines

If we worry about ipmmu_tlb_invalidate() which is called here (to perform a flush by request from P2M code, which manages a page table) and from the irq handler (to perform a flush to resume address translation), I could use a tasklet to schedule ipmmu_tlb_invalidate() from the irq handler then. This way we would get this serialized. What do you think?

I just concerned about a dead-lock issue by recursive spin locks. So, calling ipmmu_tlb_invalidate() here is OK, I think.

Best regards,
Yoshihiro Shimoda

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#55 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

In reply to Oleksandr Tyshchenko

Hi Oleksandr,

Quoted 8 lines · Oleksandr Tyshchenko
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8 -0
+static int ipmmu_iommu_domain_init(struct domain *d)
+{
+    struct ipmmu_vmsa_xen_domain *xen_domain;
+
+    xen_domain = xzalloc(struct ipmmu_vmsa_xen_domain);
+    if ( !xen_domain )
+        return -ENOMEM;
+

xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1129

Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10 -0
+    spin_lock_init(&xen_domain->lock);
+    INIT_LIST_HEAD(&xen_domain->cache_domains);
+    /*
+     * We don't create Root IPMMU domain here, it will be created on demand
+     * only, when attaching the first master device to this Xen domain in
+     * ipmmu_assign_device().
+     * xen_domain->root_domain = NULL;
+    */
+
+    dom_iommu(d)->arch.priv = xen_domain;

While looking at other part of Xen I realized you don't set IOMMU_FEAT_COHERENT_WALK. Does it mean the IOMMU walker does not support coherent walk (i.e snooping the cache)?

Note that when this feature is not set, the p2m code will require to clean each P2M entry when updated. So if the IPMMU supports coherent walk, I would strongly suggest to set the flag :).

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#56 lore raw

Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support

Quoted 1 line · Julien Grall

Hi Oleksandr,

Hi Julien.

Quoted 16 lines · Oleksandr Tyshchenko
Quoted 14 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +12 -0
+static int ipmmu_iommu_domain_init(struct domain *d)
+{
+    struct ipmmu_vmsa_xen_domain *xen_domain;
+
+    xen_domain = xzalloc(struct ipmmu_vmsa_xen_domain);
+    if ( !xen_domain )
+        return -ENOMEM;
+
+    spin_lock_init(&xen_domain->lock);
+    INIT_LIST_HEAD(&xen_domain->cache_domains);
+    /*
+     * We don't create Root IPMMU domain here, it will be created on 

demand
+     * only, when attaching the first master device to this Xen

Quoted 8 lines
Quoted 6 lines

domain in
+     * ipmmu_assign_device().
+     * xen_domain->root_domain = NULL;
+    */
+
+    dom_iommu(d)->arch.priv = xen_domain;

While looking at other part of Xen I realized you don't set IOMMU_FEAT_COHERENT_WALK. Does it mean the IOMMU walker does not support coherent walk (i.e snooping the cache)?

*AFAIK*, not supported.

Linux driver reports coherent_walk is not supported as well.

Quoted 3 lines

Note that when this feature is not set, the p2m code will require to clean each P2M entry when updated. So if the IPMMU supports coherent walk, I would strongly suggest to set the flag :).

When playing with non-shared IOMMU in Xen (two years ago), I noticed that I had forgotten to use clean_dcache after updating a page table entry. I could face faults when shattering superpages for example. Once I added it, the faults went away completely.

So, leave IOMMU_FEAT_COHERENT_WALK in disabled state, but will keep your suggestion in mind.

Quoted 3 lines

Cheers,

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#57 lore raw
In reply to Oleksandr Tyshchenko

Hello, all.

Forgot to mention that an additional patch from Xen staging is needed, otherwise Xen may crash at the early stage:
ead6b9f78355e8d366e0c80c4a73fa7fbd6d26cc
"xen/arm: cpuerrata: Align a virtual address before unmap"

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#58 lore raw
Quoted 2 lines · Oleksandr

Hello, all.

Hi,

Quoted 4 lines

Forgot to mention that an additional patch from Xen staging is needed, otherwise Xen may crash at the early stage:
ead6b9f78355e8d366e0c80c4a73fa7fbd6d26cc
"xen/arm: cpuerrata: Align a virtual address before unmap"

This patch is already merged in Xen :).

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel