Skip to contentSkip to conversation

iommu/arm: Add Renesas IPMMU-VMSA support + Linux's iommu_fwspec

PATCH Superseded Oleksandr Tyshchenko 4 participants 45 messages Sep 13, 2019 – Sep 23, 2019

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v4 cover iommu/arm: Add Renesas IPMMU-VMSA support + Linux's iommu_fwspec

From: Oleksandr Tyshchenko <[email protected]>

The purpose of this patch series is to add IPMMU-VMSA support to Xen on ARM.

Besides new IOMMU driver, this series contains "iommu_fwspec" support and new API iommu_add_dt_device() for adding DT device to IOMMU and many other things.

The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.

Please note, this driver is supposed to work only with newest R-Car Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).

----------
This driver is based on Linux's IPMMU-VMSA driver from Renesas BSP:
https://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git/tree/drivers/iommu/ipmmu-vmsa.c?h=v4.14.75-ltsi/rcar-3.9.6
and Xen's SMMU driver:
xen/drivers/passthrough/arm/smmu.c

Although Xen driver has a lot in common with Linux driver, it is not a "direct ported" copy and should be treated as such.

The major differences compare to the Linux driver are:

1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.

So, Xen driver is supposed to be used with newest R-Car Gen3 SoC revisions only (H3 ES3.0, M3-W+, etc.) which IPMMU H/W supports stage 2 translation table format.

2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.

3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.

4. IPMMU device tracking. In Xen, all IPMMU devices are managed by single driver instance. So, driver uses global list to keep track of registered devices.

----------
Series was tested on R-Car Gen3 H3 ES3.0 based boards using current staging (948a4f6 sysctl/libxl: choose a sane default for HAP)
in a system with several DMA masters being assigned to different guest domains. Guest domain reboot, destroy/create are functional.

Please note, the series depends on Julien's patch (on review):
https://lists.xenproject.org/archives/html/xen-devel/2019-08/msg01924.html

You can find the whole series (+ Julien's patch) here:
repo: https://github.com/otyshchenko1/xen.git branch: ipmmu_upstream4

You can find previous discussions here:
[V1] https://lists.xenproject.org/archives/html/xen-devel/2019-06/msg01755.html
[V2] https://lists.xenproject.org/archives/html/xen-devel/2019-08/msg00253.html
[V3] https://lists.xenproject.org/archives/html/xen-devel/2019-08/msg01948.html

Also, please note, there is a patch in ML which is intended to address the main TODO in the IPMMU-VMSA driver:
https://lists.xenproject.org/archives/html/xen-devel/2019-09/msg00973.html

Oleksandr Tyshchenko (8):
iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff iommu/arm: Add ability to handle deferred probing request iommu/arm: Order the headers alphabetically in iommu.c xen/common: Introduce _xrealloc function
xen/common: Introduce xrealloc_flex_struct() helper macros iommu/arm: Add lightweight iommu_fwspec support iommu/arm: Introduce iommu_add_dt_device API
iommu/arm: Add Renesas IPMMU-VMSA support

 xen/arch/arm/domain_build.c                 |   12 +
 xen/arch/arm/platforms/Kconfig              |    1 +
 xen/common/xmalloc_tlsf.c                   |   52 ++
 xen/drivers/passthrough/Kconfig             |   13 +
 xen/drivers/passthrough/arm/Makefile        |    3 +-
 xen/drivers/passthrough/arm/iommu.c         |  124 ++-
 xen/drivers/passthrough/arm/iommu_fwspec.c  |   93 ++
 xen/drivers/passthrough/arm/iommu_helpers.c |   84 ++
 xen/drivers/passthrough/arm/ipmmu-vmsa.c    | 1332 +++++++++++++++++++++++++++
 xen/drivers/passthrough/arm/smmu.c          |   48 +-
 xen/include/asm-arm/device.h                |    7 +-
 xen/include/asm-arm/iommu.h                 |   21 +
 xen/include/asm-arm/iommu_fwspec.h          |   68 ++
 xen/include/xen/device_tree.h               |    7 +
 xen/include/xen/iommu.h                     |   10 +
 xen/include/xen/xmalloc.h                   |   10 +
 16 files changed, 1833 insertions(+), 52 deletions(-)
 create mode 100644 xen/drivers/passthrough/arm/iommu_fwspec.c
 create mode 100644 xen/drivers/passthrough/arm/iommu_helpers.c
 create mode 100644 xen/drivers/passthrough/arm/ipmmu-vmsa.c
 create mode 100644 xen/include/asm-arm/iommu_fwspec.h
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH v4 1/8 iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff

From: Oleksandr Tyshchenko <[email protected]>

Introduce a separate file to keep various helpers which could be used by more than one IOMMU driver in order not to duplicate code.

The first candidates to be moved to the new file are SMMU driver's "map_page/unmap_page" callbacks. These callbacks neither contain any SMMU specific info nor perform any SMMU specific actions and are going to be the same across all IOMMU drivers which H/W IP shares P2M with the CPU like SMMU does.

So, move callbacks to iommu_helpers.c for the upcoming IPMMU driver to be able to re-use them.

---
Changes V3 -> V4:
- added Julien's A-b
- fixed Grammatical error
- clarified comment in a header

Changes V2 -> V3:
- put headers in alphabetical order
- retained Copyright from the SMMU code

---
 xen/drivers/passthrough/arm/Makefile        |  2 +-
 xen/drivers/passthrough/arm/iommu_helpers.c | 84 +++++++++++++++++++++++++++++
 xen/drivers/passthrough/arm/smmu.c          | 48 +----------------
 xen/include/asm-arm/iommu.h                 | 10 ++++
 4 files changed, 97 insertions(+), 47 deletions(-)
 create mode 100644 xen/drivers/passthrough/arm/iommu_helpers.c
xen/drivers/passthrough/arm/Makefile +1 -1
index b3efcfd..4abb87a 100644
@@ -1,2 +1,2 @@
-obj-y += iommu.o
+obj-y += iommu.o iommu_helpers.o
 obj-$(CONFIG_ARM_SMMU) += smmu.o
xen/drivers/passthrough/arm/iommu_helpers.c new file +84 -0

84 lines not on this page · raw message

xen/drivers/passthrough/arm/smmu.c +2 -46

60 lines not on this page · raw message

xen/include/asm-arm/iommu.h +10 -0
index 904c9ae..11dedba 100644
@@ -26,6 +26,16 @@ struct arch_iommu
 const struct iommu_ops *iommu_get_ops(void);
 void iommu_set_ops(const struct iommu_ops *ops);
 
+/*
+ * The mapping helpers below should only be used if P2M Table is shared
+ * between the CPU and the IOMMU.
+ */
+int __must_check arm_iommu_map_page(struct domain *d, dfn_t dfn, mfn_t mfn,
+                                    unsigned int flags,
+                                    unsigned int *flush_flags);
+int __must_check arm_iommu_unmap_page(struct domain *d, dfn_t dfn,
+                                      unsigned int *flush_flags);
+
 #endif /* __ARCH_ARM_IOMMU_H__ */
 
 /*
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

PATCH v4 2/8 iommu/arm: Add ability to handle deferred probing request

In reply to Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.

In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen -EAGAIN to be used for indicating that device probing is deferred.

This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.

---
Changes V3 -> V4:
- moved changes related to the "headers ordering" to a separate patch
- added explaination why domain_list is re-used
- removed the unnecessary pair of outermost parentheses when checking the list_empty

Changes V2 -> V3:
- removed deferred_probe field from struct dt_device_node, re-used domain_list instead
- documented domain_list usage
- added ASSERT to check that np->domain_list is empty before re-using it
- put deferred_probe_list to init section
- used more strict logic regarding processing devices in the deferred list
- added more comments to code
- put headers in alphabetical order

---
 xen/drivers/passthrough/arm/iommu.c | 56 +++++++++++++++++++++++++++++++++++--
 xen/include/asm-arm/device.h        |  6 +++-
 xen/include/xen/device_tree.h       |  7 +++++
 3 files changed, 66 insertions(+), 3 deletions(-)
xen/drivers/passthrough/arm/iommu.c +54 -2
index f219de9..555acfc 100644
@@ -20,6 +20,12 @@
 #include <xen/device_tree.h>
 #include <asm/device.h>
 
+/*
+ * Deferred probe list is used to keep track of devices for which driver
+ * requested deferred probing (returned -EAGAIN).
+ */
+static __initdata LIST_HEAD(deferred_probe_list);
+
 static const struct iommu_ops *iommu_ops;
 
 const struct iommu_ops *iommu_get_ops(void)
@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops)
 
 int __init iommu_hardware_setup(void)
 {
-    struct dt_device_node *np;
+    struct dt_device_node *np, *tmp;
     int rc;
     unsigned int num_iommus = 0;
 
@@ -51,6 +57,21 @@ int __init iommu_hardware_setup(void)
         rc = device_init(np, DEVICE_IOMMU, NULL);
         if ( !rc )
             num_iommus++;
+        else if ( rc == -EAGAIN )
+        {
+            /*
+             * We expect nobody uses device's domain_list at such early stage,
+             * so we can re-use it to link the device in the deferred list to
+             * avoid introducing extra list_head field in struct dt_device_node.
+             */
+            ASSERT(list_empty(&np->domain_list));
+
+            /*
+             * Driver requested deferred probing, so add this device to
+             * the deferred list for further processing.
+             */
+            list_add(&np->domain_list, &deferred_probe_list);
+        }
         /*
          * Ignore the following error codes:
          *   - EBADF: Indicate the current not is not an IOMMU
@@ -61,7 +82,38 @@ int __init iommu_hardware_setup(void)
             return rc;
     }
 
-    return ( num_iommus > 0 ) ? 0 : -ENODEV;
+    /* Return immediately if there are no initialized devices. */
+    if ( !num_iommus )
+        return list_empty(&deferred_probe_list) ? -ENODEV : -EAGAIN;
+
+    rc = 0;
+
+    /*
+     * Process devices in the deferred list if it is not empty.
+     * Check that at least one device is initialized at each loop, otherwise
+     * we may get an infinite loop. Also stop processing if we got an error
+     * other than -EAGAIN.
+     */
+    while ( !list_empty(&deferred_probe_list) && num_iommus )
+    {
+        num_iommus = 0;
+
+        list_for_each_entry_safe ( np, tmp, &deferred_probe_list, domain_list )
+        {
+            rc = device_init(np, DEVICE_IOMMU, NULL);
+            if ( !rc )
+            {
+                num_iommus++;
+
+                /* Remove initialized device from the deferred list. */
+                list_del_init(&np->domain_list);
+            }
+            else if ( rc != -EAGAIN )
+                return rc;
+        }
+    }
+
+    return rc;
 }
 
 void __hwdom_init arch_iommu_check_autotranslated_hwdom(struct domain *d)
xen/include/asm-arm/device.h +5 -1
index 63a0f36..ee1c3bc 100644
@@ -44,7 +44,11 @@ struct device_desc {
     enum device_class class;
     /* List of devices supported by this driver */
     const struct dt_device_match *dt_match;
-    /* Device initialization */
+    /*
+     * Device initialization.
+     *
+     * -EAGAIN is used to indicate that device probing is deferred.
+     */
     int (*init)(struct dt_device_node *dev, const void *data);
 };
 
xen/include/xen/device_tree.h +7 -0
index 9a7a8f2..3702e9b 100644
@@ -92,6 +92,13 @@ struct dt_device_node {
 
     /* IOMMU specific fields */
     bool is_protected;
+    /*
+     * The main purpose of this list node is to link the structure in the list
+     * of devices assigned to domain.
+     *
+     * Boot code (iommu_hardware_setup) re-uses this list to link the structure
+     * in the list of devices for which driver requested deferred probing.
+     */
     struct list_head domain_list;
 
     struct device dev;
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v4 2/8
#3 lore raw

Re: [Xen-devel] [PATCH V4 2/8] iommu/arm: Add ability to handle deferred probing request

Hi Oleksandr,

Quoted 27 lines · Oleksandr Tyshchenko
xen/drivers/passthrough/arm/iommu.c +7 -1
index f219de9..555acfc 100644
@@ -20,6 +20,12 @@
  #include <xen/device_tree.h>
  #include <asm/device.h>
 
+/*
+ * Deferred probe list is used to keep track of devices for which driver
+ * requested deferred probing (returned -EAGAIN).
+ */
+static __initdata LIST_HEAD(deferred_probe_list);
+
  static const struct iommu_ops *iommu_ops;
 
  const struct iommu_ops *iommu_get_ops(void)
@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops)
 
  int __init iommu_hardware_setup(void)
  {
-    struct dt_device_node *np;
+    struct dt_device_node *np, *tmp;
      int rc;
      unsigned int num_iommus = 0;
 

xen/drivers/passthrough/arm/iommu.c int __init iommu_hardware_setup(void) · line 63

Quoted 7 lines
xen/drivers/passthrough/arm/iommu.c +4 -0
@@ -51,6 +57,21 @@ int __init iommu_hardware_setup(void)
          rc = device_init(np, DEVICE_IOMMU, NULL);
          if ( !rc )
              num_iommus++;
+        else if ( rc == -EAGAIN )
+        {
+            /*
+             * We expect nobody uses device's domain_list at such early stage,

NIT: s/We expect nobody uses/Nobody should use/

Quoted 4 lines
xen/drivers/passthrough/arm/iommu.c +4 -0
+             * so we can re-use it to link the device in the deferred list to
+             * avoid introducing extra list_head field in struct dt_device_node.
+             */
+            ASSERT(list_empty(&np->domain_list));

[...]

Quoted 23 lines

void __hwdom_init arch_iommu_check_autotranslated_hwdom(struct domain *d)

xen/include/asm-arm/device.h +5 -1
index 63a0f36..ee1c3bc 100644
@@ -44,7 +44,11 @@ struct device_desc {
      enum device_class class;
      /* List of devices supported by this driver */
      const struct dt_device_match *dt_match;
-    /* Device initialization */
+    /*
+     * Device initialization.
+     *
+     * -EAGAIN is used to indicate that device probing is deferred.
+     */
      int (*init)(struct dt_device_node *dev, const void *data);
  };
 
xen/include/xen/device_tree.h +0 -0
index 9a7a8f2..3702e9b 100644

xen/include/xen/device_tree.h struct dt_device_node { · line 96

Quoted 5 lines
xen/include/xen/device_tree.h +2 -0
@@ -92,6 +92,13 @@ struct dt_device_node {
 
      /* IOMMU specific fields */
      bool is_protected;
+    /*
+     * The main purpose of this list node is to link the structure in the list

s/node//?

xen/include/xen/device_tree.h struct dt_device_node { · line 104

Quoted 9 lines
xen/include/xen/device_tree.h +5 -0
+     * of devices assigned to domain.
+     *
+     * Boot code (iommu_hardware_setup) re-uses this list to link the structure
+     * in the list of devices for which driver requested deferred probing.
+     */
      struct list_head domain_list;
 
      struct device dev;

With the two above addressed and pending the patch it depends on [1] is acked:

Cheers,

[1] https://lists.xenproject.org/archives/html/xen-devel/2019-08/msg01924.html

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#4 lore raw

Re: [Xen-devel] [PATCH V4 2/8] iommu/arm: Add ability to handle deferred probing request

Quoted 1 line · Julien Grall

Hi Oleksandr,

Hi, Julien.

Quoted 28 lines · Oleksandr Tyshchenko
Quoted 26 lines
xen/drivers/passthrough/arm/iommu.c +0 -0
index f219de9..555acfc 100644
@@ -20,6 +20,12 @@
   #include <xen/device_tree.h>
   #include <asm/device.h>

  +/*
+ * Deferred probe list is used to keep track of devices for which driver
+ * requested deferred probing (returned -EAGAIN).
+ */

xen/drivers/passthrough/arm/iommu.c +2 -0
+static __initdata LIST_HEAD(deferred_probe_list);
+

  static const struct iommu_ops *iommu_ops;
    const struct iommu_ops *iommu_get_ops(void)

xen/drivers/passthrough/arm/iommu.c +1 -1
@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops)
     int __init iommu_hardware_setup(void)
   {
-    struct dt_device_node *np;
+    struct dt_device_node *np, *tmp;
       int rc;
       unsigned int num_iommus = 0;
   @@ -51,6 +57,21 @@ int __init iommu_hardware_setup(void)
           rc = device_init(np, DEVICE_IOMMU, NULL);

          if ( !rc )

Quoted 8 lines
Quoted 6 lines

              num_iommus++;
+        else if ( rc == -EAGAIN )
+        {
+            /*
+             * We expect nobody uses device's domain_list at such early stage,

NIT: s/We expect nobody uses/Nobody should use/

ok

Quoted 31 lines
Quoted 7 lines

+             * so we can re-use it to link the device in the deferred list to
+             * avoid introducing extra list_head field in struct dt_device_node.
+             */
+            ASSERT(list_empty(&np->domain_list));

[...]

Quoted 21 lines

  void __hwdom_init arch_iommu_check_autotranslated_hwdom(struct domain *d)

xen/include/asm-arm/device.h +5 -1
index 63a0f36..ee1c3bc 100644
@@ -44,7 +44,11 @@ struct device_desc {
       enum device_class class;
       /* List of devices supported by this driver */
       const struct dt_device_match *dt_match;
-    /* Device initialization */
+    /*
+     * Device initialization.
+     *
+     * -EAGAIN is used to indicate that device probing is deferred.
+     */
       int (*init)(struct dt_device_node *dev, const void *data);
   };

  diff --git a/xen/include/xen/device_tree.h b/xen/include/xen/device_tree.h index 9a7a8f2..3702e9b 100644

xen/include/xen/device_tree.h +0 -0
Quoted 8 lines
Quoted 6 lines
xen/include/xen/device_tree.h +2 -0
@@ -92,6 +92,13 @@ struct dt_device_node {
         /* IOMMU specific fields */
       bool is_protected;
+    /*
+     * The main purpose of this list node is to link the structure 

in the list

s/node//?

ok

Quoted 12 lines
Quoted 11 lines

+     * of devices assigned to domain.
+     *
+     * Boot code (iommu_hardware_setup) re-uses this list to link the structure
+     * in the list of devices for which driver requested deferred probing.
+     */
      struct list_head domain_list;
        struct device dev;

Quoted 3 lines

With the two above addressed and pending the patch it depends on [1] is acked:

Thank you.

Can I do anything to speed up [1]? That patch was tested and worked fine.

Quoted 5 lines

Cheers,

[1] https://lists.xenproject.org/archives/html/xen-devel/2019-08/msg01924.html

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#5 lore raw

PATCH v4 3/8 iommu/arm: Order the headers alphabetically in iommu.c

In reply to Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

Clean up the code a bit by putting the headers in alphabetical order.

---
 xen/drivers/passthrough/arm/iommu.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
xen/drivers/passthrough/arm/iommu.c +3 -2
index 555acfc..5a3d1d5 100644
@@ -15,9 +15,10 @@
  * GNU General Public License for more details.
  */
 
-#include <xen/lib.h>
-#include <xen/iommu.h>
 #include <xen/device_tree.h>
+#include <xen/iommu.h>
+#include <xen/lib.h>
+
 #include <asm/device.h>
 
 /*
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Acked-by on v4 3/8
#6 lore raw

Re: [Xen-devel] [PATCH V4 3/8] iommu/arm: Order the headers alphabetically in iommu.c

Hi Oleksandr,

Quoted 5 lines · Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

Clean up the code a bit by putting the headers in alphabetical order.

And committed.

Cheers,

Quoted 22 lines
---
  xen/drivers/passthrough/arm/iommu.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
xen/drivers/passthrough/arm/iommu.c +3 -2
index 555acfc..5a3d1d5 100644
@@ -15,9 +15,10 @@
   * GNU General Public License for more details.
   */
 
-#include <xen/lib.h>
-#include <xen/iommu.h>
  #include <xen/device_tree.h>
+#include <xen/iommu.h>
+#include <xen/lib.h>
+
  #include <asm/device.h>
 
  /*
Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#7 lore raw

PATCH v4 4/8 xen/common: Introduce _xrealloc function

In reply to Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

This patch introduces type-unsafe function which besides re-allocation handles the following corner cases:
1. if requested size is zero, it will behave like xfree
2. if incoming pointer is not valid (NULL or ZERO_BLOCK_PTR), it will behave like xmalloc

If both pointer and size are valid the function will re-allocate and copy only if requested size and alignment don't fit in already allocated space.

Subsequent patch will add type-safe helper macros.

CC Andrew Cooper <[email protected]>
CC George Dunlap <[email protected]>
CC Ian Jackson <[email protected]>
CC Jan Beulich <[email protected]>
CC Julien Grall <[email protected]>
CC Konrad Rzeszutek Wilk <[email protected]>
CC Stefano Stabellini <[email protected]>
CC Tim Deegan <[email protected]>
CC Wei Liu <[email protected]>
CC Paul Durrant <[email protected]>

---
Changes V3 -> V4:
- add check for the alignment compatibility
- properly detect current size (take into the account a possible fake alignment header)
- update comment in code/patch description

Changes RFC -> V3:
- behave like xmalloc if incoming pointer is ZERO_BLOCK_PTR or NULL
- return ZERO_BLOCK_PTR after xfree if requested size is zero
- add patch description
- use allocator internals to recognize current size of the incoming pointer
- do not re-allocate and copy if requested size fits in already allocated space

...

Original patch was initially posted by Sameer Goel:
https://lists.xen.org/archives/html/xen-devel/2017-06/msg00858.html

This could be considered as another attempt to add it:
https://www.mail-archive.com/[email protected]/msg21335.html

[As it was previously discussed with Julien in IRC]

The reason for this patch to be an RFC is that patch itself is not completely correct and I don't fully understand what/how should be done for this patch to be accepted. Or whether community even wants this to go in. So, to avoid bike shedding, the first target is to collect feedback.

For everyone who wants more details why this is needed and where used, please see next patch of this thread:
"iommu/arm: Add lightweight iommu_fwspec support"

In a nutshell, the upcoming "iommu_fwspec" support on ARM is going to use xrealloc to expand an array for device IDs. We really want to have "iommu_fwspec" support which will give us a generic abstract way to add new device to the IOMMU based on the generic IOMMU DT binding.

---
 xen/common/xmalloc_tlsf.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++
 xen/include/xen/xmalloc.h |  1 +
 2 files changed, 53 insertions(+)
xen/common/xmalloc_tlsf.c +52 -0
index e98ad65..2b240b1 100644
@@ -598,6 +598,58 @@ void *_xzalloc(unsigned long size, unsigned long align)
     return p ? memset(p, 0, size) : p;
 }
 
+void *_xrealloc(void *ptr, unsigned long size, unsigned long align)
+{
+    unsigned long curr_size, tmp_size;
+    void *p;
+
+    if ( !size )
+    {
+        xfree(ptr);
+        return ZERO_BLOCK_PTR;
+    }
+
+    if ( ptr == NULL || ptr == ZERO_BLOCK_PTR )
+        return _xmalloc(size, align);
+
+    if ( !((unsigned long)ptr & (PAGE_SIZE - 1)) )
+        curr_size = PFN_ORDER(virt_to_page(ptr)) << PAGE_SHIFT;
+    else
+    {
+        struct bhdr *b = (struct bhdr *)((char *)ptr - BHDR_OVERHEAD);
+
+        if ( b->size & FREE_BLOCK )
+        {
+            p = (char *)ptr - (b->size & ~FREE_BLOCK);
+            b = (struct bhdr *)((char *)p - BHDR_OVERHEAD);
+            ASSERT(!(b->size & FREE_BLOCK));
+        }
+
+        curr_size = b->size & BLOCK_SIZE_MASK;
+    }
+
+    ASSERT((align & (align - 1)) == 0);
+    if ( align < MEM_ALIGN )
+        align = MEM_ALIGN;
+    tmp_size = size + align - MEM_ALIGN;
+
+    if ( tmp_size < PAGE_SIZE )
+        tmp_size = ( tmp_size < MIN_BLOCK_SIZE ) ? MIN_BLOCK_SIZE :
+            ROUNDUP_SIZE(tmp_size);
+
+    if ( tmp_size <= curr_size && ((unsigned long)ptr & (align - 1)) == 0 )
+        return ptr; /* the size and alignment fit in already allocated space */
+
+    p = _xmalloc(size, align);
+    if ( p )
+    {
+        memcpy(p, ptr, min(curr_size, size));
+        xfree(ptr);
+    }
+
+    return p;
+}
+
 void xfree(void *p)
 {
     struct bhdr *b;
xen/include/xen/xmalloc.h +1 -0
index f075d2d..831152f 100644
@@ -51,6 +51,7 @@ extern void xfree(void *);
 /* Underlying functions */
 extern void *_xmalloc(unsigned long size, unsigned long align);
 extern void *_xzalloc(unsigned long size, unsigned long align);
+extern void *_xrealloc(void *ptr, unsigned long size, unsigned long align);
 
 static inline void *_xmalloc_array(
     unsigned long size, unsigned long align, unsigned long num)
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#8 lore raw

Re: [Xen-devel] [PATCH V4 4/8] xen/common: Introduce _xrealloc function

Quoted 12 lines · Oleksandr Tyshchenko
xen/common/xmalloc_tlsf.c +6 -0
@@ -598,6 +598,58 @@ void *_xzalloc(unsigned long size, unsigned long align)
     return p ? memset(p, 0, size) : p;
 }
 
+void *_xrealloc(void *ptr, unsigned long size, unsigned long align)
+{
+    unsigned long curr_size, tmp_size;
+    void *p;
+
+    if ( !size )

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 616

Quoted 10 lines
xen/common/xmalloc_tlsf.c +10 -0
+    {
+        xfree(ptr);
+        return ZERO_BLOCK_PTR;
+    }
+
+    if ( ptr == NULL || ptr == ZERO_BLOCK_PTR )
+        return _xmalloc(size, align);
+
+    if ( !((unsigned long)ptr & (PAGE_SIZE - 1)) )
+        curr_size = PFN_ORDER(virt_to_page(ptr)) << PAGE_SHIFT;

While the present MAX_ORDER setting will prevent allocations of 4GiB or above from succeeding, may I ask that you don't introduce latent issues in case MAX_ORDER would ever need bumping?

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 626

Quoted 10 lines
xen/common/xmalloc_tlsf.c +10 -0
+    else
+    {
+        struct bhdr *b = (struct bhdr *)((char *)ptr - BHDR_OVERHEAD);
+
+        if ( b->size & FREE_BLOCK )
+        {
+            p = (char *)ptr - (b->size & ~FREE_BLOCK);
+            b = (struct bhdr *)((char *)p - BHDR_OVERHEAD);
+            ASSERT(!(b->size & FREE_BLOCK));
+        }

This matches the respective xfree() code fragment, and needs to remain in sync. Which suggests introducing a helper function instead of duplicating the code. And please omit the unnecessary casts to char *.

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 628

Quoted 1 line
xen/common/xmalloc_tlsf.c +1 -0
+        curr_size = b->size & BLOCK_SIZE_MASK;

_xmalloc() has

        b->size = pad | FREE_BLOCK;

i.e. aiui what you calculate above is the padding size, not the overall block size.

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 637

Quoted 9 lines
xen/common/xmalloc_tlsf.c +9 -0
+    }
+
+    ASSERT((align & (align - 1)) == 0);
+    if ( align < MEM_ALIGN )
+        align = MEM_ALIGN;
+    tmp_size = size + align - MEM_ALIGN;
+
+    if ( tmp_size < PAGE_SIZE )
+        tmp_size = ( tmp_size < MIN_BLOCK_SIZE ) ? MIN_BLOCK_SIZE :

Stray blanks inside parentheses.

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 641

Quoted 4 lines
xen/common/xmalloc_tlsf.c +4 -0
+            ROUNDUP_SIZE(tmp_size);
+
+    if ( tmp_size <= curr_size && ((unsigned long)ptr & (align - 1)) == 0 )
+        return ptr; /* the size and alignment fit in already allocated space */

You also don't seem to ever update ptr in case you want to use the (head) padding, i.e. you'd hand back a pointer to a block which the caller would assume extends past its actual end. I think you want to calculate the new tentative pointer (taking the requested alignment into account), and only from that calculate curr_size (which perhaps would better be named "usable" or "space" or some such). Obviously the (head) padding block may need updating, too.

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 651

Quoted 9 lines
xen/common/xmalloc_tlsf.c +9 -0
+    p = _xmalloc(size, align);
+    if ( p )
+    {
+        memcpy(p, ptr, min(curr_size, size));
+        xfree(ptr);
+    }
+
+    return p;
+}

As a final remark - did you consider zero(?)-filling the tail portion? While C's realloc() isn't specified to do so, since there's no (not going to be a) zeroing counterpart, doing so may be safer overall.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#9 lore raw

Re: [Xen-devel] [PATCH V4 4/8] xen/common: Introduce _xrealloc function

On 16.09.19 13:13, Jan Beulich wrote:

Hi, Jan

Quoted 18 lines · Oleksandr Tyshchenko
Quoted 17 lines
xen/common/xmalloc_tlsf.c +11 -0
@@ -598,6 +598,58 @@ void *_xzalloc(unsigned long size, unsigned long align)
      return p ? memset(p, 0, size) : p;
  }
 
+void *_xrealloc(void *ptr, unsigned long size, unsigned long align)
+{
+    unsigned long curr_size, tmp_size;
+    void *p;
+
+    if ( !size )
+    {
+        xfree(ptr);
+        return ZERO_BLOCK_PTR;
+    }
+
Quoted 8 lines

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 616

Quoted 5 lines
xen/common/xmalloc_tlsf.c +5 -0
+    if ( ptr == NULL || ptr == ZERO_BLOCK_PTR )
+        return _xmalloc(size, align);
+
+    if ( !((unsigned long)ptr & (PAGE_SIZE - 1)) )
+        curr_size = PFN_ORDER(virt_to_page(ptr)) << PAGE_SHIFT;

While the present MAX_ORDER setting will prevent allocations of 4GiB or above from succeeding, may I ask that you don't introduce latent issues in case MAX_ORDER would ever need bumping?

Sure (I assume, you are talking about possible truncation):

if ( !((unsigned long)ptr & (PAGE_SIZE - 1)) )
    curr_size = (unsigned long)PFN_ORDER(virt_to_page(ptr)) << PAGE_SHIFT;

Quoted 11 lines

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 626

Quoted 11 lines
xen/common/xmalloc_tlsf.c +10 -0
+    else
+    {
+        struct bhdr *b = (struct bhdr *)((char *)ptr - BHDR_OVERHEAD);
+
+        if ( b->size & FREE_BLOCK )
+        {
+            p = (char *)ptr - (b->size & ~FREE_BLOCK);
+            b = (struct bhdr *)((char *)p - BHDR_OVERHEAD);
+            ASSERT(!(b->size & FREE_BLOCK));
+        }
Quoted 4 lines

This matches the respective xfree() code fragment, and needs to remain in sync. Which suggests introducing a helper function instead of duplicating the code. And please omit the unnecessary casts to char *.

Sounds reasonable, will do.

Quoted 8 lines

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 628

Quoted 2 lines
xen/common/xmalloc_tlsf.c +1 -0
+        curr_size = b->size & BLOCK_SIZE_MASK;

_xmalloc() has

         b->size = pad | FREE_BLOCK;

i.e. aiui what you calculate above is the padding size, not the overall block size.

I have to admit that I am not familiar with allocator internals enough, but

I meant to calculate overall block size (the alignment padding is stripped if present)...

Quoted 10 lines

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 637

Quoted 9 lines
xen/common/xmalloc_tlsf.c +9 -0
+    }
+
+    ASSERT((align & (align - 1)) == 0);
+    if ( align < MEM_ALIGN )
+        align = MEM_ALIGN;
+    tmp_size = size + align - MEM_ALIGN;
+
+    if ( tmp_size < PAGE_SIZE )
+        tmp_size = ( tmp_size < MIN_BLOCK_SIZE ) ? MIN_BLOCK_SIZE :

Stray blanks inside parentheses.

ok

Quoted 5 lines

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 641

Quoted 5 lines
xen/common/xmalloc_tlsf.c +4 -0
+            ROUNDUP_SIZE(tmp_size);
+
+    if ( tmp_size <= curr_size && ((unsigned long)ptr & (align - 1)) == 0 )
+        return ptr; /* the size and alignment fit in already allocated space */
Quoted 7 lines

You also don't seem to ever update ptr in case you want to use the (head) padding, i.e. you'd hand back a pointer to a block which the caller would assume extends past its actual end. I think you want to calculate the new tentative pointer (taking the requested alignment into account), and only from that calculate curr_size (which perhaps would better be named "usable" or "space" or some such). Obviously the (head) padding block may need updating, too.

I am afraid I don't completely understand your point here. And sorry for the maybe naive question, but what is the "(head) padding" here?

Quoted 5 lines
Quoted 5 lines
xen/common/xmalloc_tlsf.c +5 -0
+    p = _xmalloc(size, align);
+    if ( p )
+    {
+        memcpy(p, ptr, min(curr_size, size));
+        xfree(ptr);
Quoted 8 lines

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 651

Quoted 4 lines
xen/common/xmalloc_tlsf.c +4 -0
+    }
+
+    return p;
+}

As a final remark - did you consider zero(?)-filling the tail portion? While C's realloc() isn't specified to do so, since there's no (not going to be a) zeroing counterpart, doing so may be safer overall.

Probably, worth doing. Will zero it.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#10 lore raw

Re: [Xen-devel] [PATCH V4 4/8] xen/common: Introduce _xrealloc function

Quoted 27 lines · Oleksandr
Quoted 18 lines · Jan Beulich
Quoted 17 lines · Oleksandr Tyshchenko
xen/common/xmalloc_tlsf.c +11 -0
@@ -598,6 +598,58 @@ void *_xzalloc(unsigned long size, unsigned long align)
      return p ? memset(p, 0, size) : p;
  }
 
+void *_xrealloc(void *ptr, unsigned long size, unsigned long align)
+{
+    unsigned long curr_size, tmp_size;
+    void *p;
+
+    if ( !size )
+    {
+        xfree(ptr);
+        return ZERO_BLOCK_PTR;
+    }
+
Quoted 8 lines

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 616

Quoted 5 lines
xen/common/xmalloc_tlsf.c +5 -0
+    if ( ptr == NULL || ptr == ZERO_BLOCK_PTR )
+        return _xmalloc(size, align);
+
+    if ( !((unsigned long)ptr & (PAGE_SIZE - 1)) )
+        curr_size = PFN_ORDER(virt_to_page(ptr)) << PAGE_SHIFT;

While the present MAX_ORDER setting will prevent allocations of 4GiB or above from succeeding, may I ask that you don't introduce latent issues in case MAX_ORDER would ever need bumping?

Quoted 4 lines

Sure (I assume, you are talking about possible truncation):

if ( !((unsigned long)ptr & (PAGE_SIZE - 1)) )
    curr_size = (unsigned long)PFN_ORDER(virt_to_page(ptr)) << PAGE_SHIFT;

Yes.

Quoted 12 lines
Quoted 11 lines

xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 641

Quoted 4 lines
xen/common/xmalloc_tlsf.c +4 -0
+            ROUNDUP_SIZE(tmp_size);
+
+    if ( tmp_size <= curr_size && ((unsigned long)ptr & (align - 1)) == 0 )
+        return ptr; /* the size and alignment fit in already allocated space */

You also don't seem to ever update ptr in case you want to use the (head) padding, i.e. you'd hand back a pointer to a block which the caller would assume extends past its actual end. I think you want to calculate the new tentative pointer (taking the requested alignment into account), and only from that calculate curr_size (which perhaps would better be named "usable" or "space" or some such). Obviously the (head) padding block may need updating, too.

Quoted 2 lines

I am afraid I don't completely understand your point here. And sorry for the maybe naive question, but what is the "(head) padding" here?

The very padding talked about earlier. I did add "(head)" to clarify it's that specific case - after all tail padding is far more common.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#11 lore raw

Re: [Xen-devel] [PATCH V4 4/8] xen/common: Introduce _xrealloc function

On 16.09.19 18:24, Jan Beulich wrote:

Hi, Jan.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+            ROUNDUP_SIZE(tmp_size);
+
+    if ( tmp_size <= curr_size && ((unsigned long)ptr & (align - 1)) == 0 )
+        return ptr; /* the size and alignment fit in already allocated space */

You also don't seem to ever update ptr in case you want to use the (head) padding, i.e. you'd hand back a pointer to a block which the caller would assume extends past its actual end. I think you want to calculate the new tentative pointer (taking the requested alignment into account), and only from that calculate curr_size (which perhaps would better be named "usable" or "space" or some such). Obviously the (head) padding block may need updating, too.

Quoted 4 lines
Quoted 2 lines

I am afraid I don't completely understand your point here. And sorry for the maybe naive question, but what is the "(head) padding" here?

The very padding talked about earlier. I did add "(head)" to clarify it's that specific case - after all tail padding is far more common.

Still unsure, I completely understand your point regarding calculating tentative pointer and then curr_size.

----------

Does the diff below is close to what you meant?

---
 xen/common/xmalloc_tlsf.c | 113 ++++++++++++++++++++++++++++++++++++++-------- xen/include/xen/xmalloc.h |   1 +

 2 files changed, 96 insertions(+), 18 deletions(-)
xen/common/xmalloc_tlsf.c +0 -0
index e98ad65..f24c97c 100644
@@ -554,10 +554,40 @@ static void tlsf_init(void)

 #define ZERO_BLOCK_PTR ((void *)-1L)
 #endif

+static void *strip_padding(void *p)
+{
+    struct bhdr *b = (struct bhdr *)(p - BHDR_OVERHEAD); +
+    if ( b->size & FREE_BLOCK )
+    {
+        p -= b->size & ~FREE_BLOCK;
+        b = (struct bhdr *)(p - BHDR_OVERHEAD);
+        ASSERT(!(b->size & FREE_BLOCK));
+    }
+
+    return p;
+}
+
+static void *add_padding(void *p, unsigned long align) +{
+    u32 pad;
+
+    if ( (pad = -(long)p & (align - 1)) != 0 )
+    {
+        char *q = (char *)p + pad;
+        struct bhdr *b = (struct bhdr *)(q - BHDR_OVERHEAD); +
+        ASSERT(q > (char *)p);
+        b->size = pad | FREE_BLOCK;
+        p = q;
+    }
+
+    return p;
+}
+
 void *_xmalloc(unsigned long size, unsigned long align) {
     void *p = NULL;
-    u32 pad;

     ASSERT(!in_irq());

@@ -578,14 +608,7 @@ void *_xmalloc(unsigned long size, unsigned long align) return xmalloc_whole_pages(size - align + MEM_ALIGN, align);

     /* Add alignment padding. */
-    if ( (pad = -(long)p & (align - 1)) != 0 )
-    {
-        char *q = (char *)p + pad;
-        struct bhdr *b = (struct bhdr *)(q - BHDR_OVERHEAD); -        ASSERT(q > (char *)p);
-        b->size = pad | FREE_BLOCK;
-        p = q;
-    }
+    p = add_padding(p, align);

     ASSERT(((unsigned long)p & (align - 1)) == 0);
     return p;
@@ -598,10 +621,70 @@ void *_xzalloc(unsigned long size, unsigned long align) return p ? memset(p, 0, size) : p;
 }

-void xfree(void *p)
+void *_xrealloc(void *ptr, unsigned long size, unsigned long align) {
-    struct bhdr *b;
+    unsigned long curr_size, tmp_size;
+    void *p;
+
+    if ( !size )
+    {
+        xfree(ptr);
+        return ZERO_BLOCK_PTR;
+    }
+
+    if ( ptr == NULL || ptr == ZERO_BLOCK_PTR )
+        return _xmalloc(size, align);
+
+    ASSERT((align & (align - 1)) == 0);
+    if ( align < MEM_ALIGN )
+        align = MEM_ALIGN;
+
+    tmp_size = size + align - MEM_ALIGN;
+
+    if ( tmp_size < PAGE_SIZE )
+        tmp_size = (tmp_size < MIN_BLOCK_SIZE) ? MIN_BLOCK_SIZE :
+            ROUNDUP_SIZE(tmp_size);
+
+    if ( !((unsigned long)ptr & (PAGE_SIZE - 1)) )
+    {
+        curr_size = (unsigned long)PFN_ORDER(virt_to_page(ptr)) << PAGE_SHIFT; +
+        if ( tmp_size <= curr_size && ((unsigned long)ptr & (align - 1)) == 0 ) +            return ptr;
+    }
+    else
+    {
+        struct bhdr *b;
+
+        /* Strip alignment padding. */
+        p = strip_padding(ptr);
+
+        b = (struct bhdr *)(p - BHDR_OVERHEAD);
+        curr_size = b->size & BLOCK_SIZE_MASK;
+
+        if ( tmp_size <= curr_size )
+        {
+            /* Add alignment padding. */
+            p = add_padding(p, align);
+
+            ASSERT(((unsigned long)p & (align - 1)) == 0);
+
+            return p;
+        }
+    }
+
+    p = _xzalloc(size, align);
+    if ( p )
+    {
+        memcpy(p, ptr, min(curr_size, size));
+        xfree(ptr);
+    }
+
+    return p;
+}
+
+void xfree(void *p)
+{
     if ( p == NULL || p == ZERO_BLOCK_PTR )
         return;

@@ -626,13 +709,7 @@ void xfree(void *p)
     }

     /* Strip alignment padding. */
-    b = (struct bhdr *)((char *)p - BHDR_OVERHEAD); -    if ( b->size & FREE_BLOCK )
-    {
-        p = (char *)p - (b->size & ~FREE_BLOCK); -        b = (struct bhdr *)((char *)p - BHDR_OVERHEAD); -        ASSERT(!(b->size & FREE_BLOCK));
-    }
+    p = strip_padding(p);

     xmem_pool_free(p, xenpool);
 }

xen/include/xen/xmalloc.h +0 -0
index f075d2d..831152f 100644
@@ -51,6 +51,7 @@ extern void xfree(void *);

 /* Underlying functions */
 extern void *_xmalloc(unsigned long size, unsigned long align); extern void *_xzalloc(unsigned long size, unsigned long align); +extern void *_xrealloc(void *ptr, unsigned long size, unsigned long align);

 static inline void *_xmalloc_array(
     unsigned long size, unsigned long align, unsigned long num)

Signature · 13 lines
2.7.4


-- 
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#12 lore raw

Re: [Xen-devel] [PATCH V4 4/8] xen/common: Introduce _xrealloc function

Quoted 1 line · Oleksandr

Does the diff below is close to what you meant?

Almost.

Quoted 25 lines
(quoted diff) +18 -2
@@ -598,10 +621,70 @@ void *_xzalloc(unsigned long size, unsigned long align)
      return p ? memset(p, 0, size) : p;
  }
 
-void xfree(void *p)
+void *_xrealloc(void *ptr, unsigned long size, unsigned long align)
  {
-    struct bhdr *b;
+    unsigned long curr_size, tmp_size;
+    void *p;
+
+    if ( !size )
+    {
+        xfree(ptr);
+        return ZERO_BLOCK_PTR;
+    }
+
+    if ( ptr == NULL || ptr == ZERO_BLOCK_PTR )
+        return _xmalloc(size, align);
+
+    ASSERT((align & (align - 1)) == 0);
+    if ( align < MEM_ALIGN )
+        align = MEM_ALIGN;
+
+    tmp_size = size + align - MEM_ALIGN;

Review void *_xzalloc(unsigned long size, unsigned long align) · line 652

Quoted 10 lines
(quoted diff) +10 -0
+
+    if ( tmp_size < PAGE_SIZE )
+        tmp_size = (tmp_size < MIN_BLOCK_SIZE) ? MIN_BLOCK_SIZE :
+            ROUNDUP_SIZE(tmp_size);
+
+    if ( !((unsigned long)ptr & (PAGE_SIZE - 1)) )
+    {
+        curr_size = (unsigned long)PFN_ORDER(virt_to_page(ptr)) << PAGE_SHIFT;
+
+        if ( tmp_size <= curr_size && ((unsigned long)ptr & (align - 1)) == 0 )

You mean "size" here I think, not "tmp_size". See how xmalloc_whole_pages() gets called from _xmalloc() with an "adjusted back" value.

And as said, please clean up the code you move or add anew: Use casts only where really needed, transform types to appropriate "modern" ones, etc.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#13 lore raw

Re: [Xen-devel] [PATCH V4 4/8] xen/common: Introduce _xrealloc function

On 23.09.19 16:31, Jan Beulich wrote:

Hi, Jan

Quoted 14 lines
Quoted 14 lines
(quoted diff) +13 -0
+
+    if ( ptr == NULL || ptr == ZERO_BLOCK_PTR )
+        return _xmalloc(size, align);
+
+    ASSERT((align & (align - 1)) == 0);
+    if ( align < MEM_ALIGN )
+        align = MEM_ALIGN;
+
+    tmp_size = size + align - MEM_ALIGN;
+
+    if ( tmp_size < PAGE_SIZE )
+        tmp_size = (tmp_size < MIN_BLOCK_SIZE) ? MIN_BLOCK_SIZE :
+            ROUNDUP_SIZE(tmp_size);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+    if ( !((unsigned long)ptr & (PAGE_SIZE - 1)) )
+    {
+        curr_size = (unsigned long)PFN_ORDER(virt_to_page(ptr)) << PAGE_SHIFT;
+
+        if ( tmp_size <= curr_size && ((unsigned long)ptr & (align - 1)) == 0 )

You mean "size" here I think, not "tmp_size". See how xmalloc_whole_pages() gets called from _xmalloc() with an "adjusted back" value.

Yes, thank you for pointing this.

Quoted 3 lines

And as said, please clean up the code you move or add anew: Use casts only where really needed, transform types to appropriate "modern" ones, etc.

ok, will double check.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#14 lore raw

PATCH v4 5/8 xen/common: Introduce xrealloc_flex_struct() helper macros

In reply to Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

This patch introduces type-safe helper macros to re-allocate space for a structure with a flexible array of typed objects.

For example, if we need to re-size an array with a single element:

   struct arrlen
   {
      size_t len;
      int data[1];
   };

We can use the proposed macros in the following way:

   new_ptr = realloc_flex_struct(old_ptr, data, nr_elem);

Subsequent patch will use this macros.

Also, while here, introduce xmalloc_flex_struct() to allocate space for a structure with a flexible array of typed objects.

CC Andrew Cooper <[email protected]>
CC George Dunlap <[email protected]>
CC Ian Jackson <[email protected]>
CC Jan Beulich <[email protected]>
CC Julien Grall <[email protected]>
CC Konrad Rzeszutek Wilk <[email protected]>
CC Stefano Stabellini <[email protected]>
CC Tim Deegan <[email protected]>
CC Wei Liu <[email protected]>
CC Paul Durrant <[email protected]>

---
Changes V3 -> V4:
- clarified patch description
- modified to not use leading underscores
- removed unnecessary pair of outermost parentheses
- modified to use "nr" instead of "len"
- placed xmalloc_flex_struct before xrealloc_flex_struct
- simplified xrealloc_flex_struct macros

---
 xen/include/xen/xmalloc.h | 9 +++++++++
 1 file changed, 9 insertions(+)
xen/include/xen/xmalloc.h +9 -0
index 831152f..f0736ce 100644
@@ -35,6 +35,15 @@
 #define xzalloc_array(_type, _num) \
     ((_type *)_xzalloc_array(sizeof(_type), __alignof__(_type), _num))
 
+/* Allocate space for a structure with a flexible array of typed objects. */
+#define xmalloc_flex_struct(type, field, nr) \
+    (type *)_xmalloc(offsetof(type, field[nr]), __alignof__(type))
+
+/* Re-allocate space for a structure with a flexible array of typed objects. */
+#define xrealloc_flex_struct(ptr, field, nr)                          \
+    (typeof(ptr))_xrealloc(ptr, offsetof(typeof(*(ptr)), field[nr]),  \
+                           __alignof__(typeof(*(ptr))))
+
 /* Allocate untyped storage. */
 #define xmalloc_bytes(_bytes) _xmalloc(_bytes, SMP_CACHE_BYTES)
 #define xzalloc_bytes(_bytes) _xzalloc(_bytes, SMP_CACHE_BYTES)
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v4 5/8
#15 lore raw

Re: [Xen-devel] [PATCH V4 5/8] xen/common: Introduce xrealloc_flex_struct() helper macros

Quoted 4 lines · Oleksandr Tyshchenko
xen/include/xen/xmalloc.h +0 -0
@@ -35,6 +35,15 @@
 #define xzalloc_array(_type, _num) \

xen/include/xen/xmalloc.h line 45

Quoted 10 lines
xen/include/xen/xmalloc.h +8 -0
     ((_type *)_xzalloc_array(sizeof(_type), __alignof__(_type), _num))
 
+/* Allocate space for a structure with a flexible array of typed objects. */
+#define xmalloc_flex_struct(type, field, nr) \
+    (type *)_xmalloc(offsetof(type, field[nr]), __alignof__(type))
+
+/* Re-allocate space for a structure with a flexible array of typed objects. */
+#define xrealloc_flex_struct(ptr, field, nr)                          \
+    (typeof(ptr))_xrealloc(ptr, offsetof(typeof(*(ptr)), field[nr]),  \
+                           __alignof__(typeof(*(ptr))))

With the missing parentheses around the entire constructs added

I'd like to note though that it sort of feels as if this notation isn't going to provide maximum flexibility. I therefore wonder whether the last two parameters shouldn't be combined, resulting in an invocation like

    ptr = xmalloc_flex_struct(struct s, field[5]);

But I realize this would allow for (more; I'll reply to patch 6 in a minute) abuse, so this wouldn't be a clear win.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#16 lore raw

Re: [Xen-devel] [PATCH V4 5/8] xen/common: Introduce xrealloc_flex_struct() helper macros

On 16.09.19 13:37, Jan Beulich wrote:

Hi, Jan

Quoted 16 lines · Oleksandr Tyshchenko
Quoted 4 lines
xen/include/xen/xmalloc.h +0 -0
@@ -35,6 +35,15 @@
  #define xzalloc_array(_type, _num) \

xen/include/xen/xmalloc.h line 45

Quoted 10 lines
xen/include/xen/xmalloc.h +8 -0
      ((_type *)_xzalloc_array(sizeof(_type), __alignof__(_type), _num))
 
+/* Allocate space for a structure with a flexible array of typed objects. */
+#define xmalloc_flex_struct(type, field, nr) \
+    (type *)_xmalloc(offsetof(type, field[nr]), __alignof__(type))
+
+/* Re-allocate space for a structure with a flexible array of typed objects. */
+#define xrealloc_flex_struct(ptr, field, nr)                          \
+    (typeof(ptr))_xrealloc(ptr, offsetof(typeof(*(ptr)), field[nr]),  \
+                           __alignof__(typeof(*(ptr))))

With the missing parentheses around the entire constructs added

Quoted 1 line

Thank you.

Quoted 9 lines

I'd like to note though that it sort of feels as if this notation isn't going to provide maximum flexibility. I therefore wonder whether the last two parameters shouldn't be combined, resulting in an invocation like

     ptr = xmalloc_flex_struct(struct s, field[5]);

But I realize this would allow for (more; I'll reply to patch 6 in a minute) abuse, so this wouldn't be a clear win.

Agree.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#17 lore raw

Re: [Xen-devel] [PATCH V4 5/8] xen/common: Introduce xrealloc_flex_struct() helper macros

Hi Jan

Quoted 21 lines
Quoted 19 lines · Oleksandr Tyshchenko
Quoted 10 lines
xen/include/xen/xmalloc.h +0 -0
@@ -35,6 +35,15 @@
   #define xzalloc_array(_type, _num) \
       ((_type *)_xzalloc_array(sizeof(_type), __alignof__(_type), _num))

  +/* Allocate space for a structure with a flexible array of typed objects. */

xen/include/xen/xmalloc.h +4 -0
+#define xmalloc_flex_struct(type, field, nr) \
+    (type *)_xmalloc(offsetof(type, field[nr]), __alignof__(type))
+
+/* Re-allocate space for a structure with a flexible array of typed 
Quoted 6 lines

objects. */
+#define xrealloc_flex_struct(ptr, field,
nr)                          \
+    (typeof(ptr))_xrealloc(ptr, offsetof(typeof(*(ptr)), field[nr]),  \
+                           __alignof__(typeof(*(ptr))))

With the missing parentheses around the entire constructs added

Quoted 1 line
Quoted 1 line

Thank you.

Would you be happy if I add xzalloc_flex_struct here as well (may I retain your R-b)?

Actually the xzalloc_flex_struct better fits in [1] ...

[1] https://www.mail-archive.com/[email protected]/msg55557.html

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#18 lore raw

Re: [Xen-devel] [PATCH V4 5/8] xen/common: Introduce xrealloc_flex_struct() helper macros

Quoted 22 lines · Oleksandr
Quoted 22 lines
Quoted 21 lines · Oleksandr Tyshchenko
Quoted 12 lines
xen/include/xen/xmalloc.h +0 -0
@@ -35,6 +35,15 @@
   #define xzalloc_array(_type, _num) \
       ((_type *)_xzalloc_array(sizeof(_type), __alignof__(_type), 
 _num))

  +/* Allocate space for a structure with a flexible array of typed objects. */

xen/include/xen/xmalloc.h +4 -0
+#define xmalloc_flex_struct(type, field, nr) \
+    (type *)_xmalloc(offsetof(type, field[nr]), __alignof__(type))
+
+/* Re-allocate space for a structure with a flexible array of typed 
Quoted 6 lines

objects. */
+#define xrealloc_flex_struct(ptr, field,
nr)                          \
+    (typeof(ptr))_xrealloc(ptr, offsetof(typeof(*(ptr)), field[nr]),  \
+                           __alignof__(typeof(*(ptr))))

With the missing parentheses around the entire constructs added

Quoted 5 lines
Quoted 1 line

Thank you.

Would you be happy if I add xzalloc_flex_struct here as well (may I retain your R-b)?

Yes to both.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#19 lore raw

PATCH v4 6/8 iommu/arm: Add lightweight iommu_fwspec support

In reply to Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

We need to have some abstract way to add new device to the IOMMU based on the generic IOMMU DT bindings [1] which can be used for both DT (right now) and ACPI (in future).

For that reason we can borrow the idea used in Linux these days called "iommu_fwspec". Having this in, it will be possible to configure IOMMU master interfaces of the device (device IDs) from a single common place and avoid keeping almost identical look-up implementations in each IOMMU driver.

There is no need to port the whole implementation of "iommu_fwspec" to Xen, we could, probably, end up with a much simpler solution, some "stripped down" version which fits our requirements.

So, this patch adds the following:
1. A common structure "iommu_fwspec" to hold the the per-device firmware data
2. New member "iommu_fwspec" of struct device
3. Functions/helpers to deal with "dev->iommu_fwspec"

It should be noted that in comparison of the original "iommu_fwspec" Xen's variant doesn't contain some fields, which are not really needed at the moment (ops, flag) and "iommu_fwnode" field was replaced by "iommu_dev" to avoid porting a lot of code (to support "fwnode_handle") with little benefit.

The "iommu_fwspec" support is based on the Linux's commit:
f74c2bb98776e2de508f4d607cd519873065118e "Linux 5.3-rc8"

Subsequent patches will use of that support.

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt

---
Changes V3 -> V4:
- modified iommu_fwspec_add_ids() to use new implementation of xrealloc_flex_struct()
- mentioned exact Linux version we are based on
- fixed Grammatical error

Changes V2 -> V3:
- added Copyright from Linux
- ordered the headers alphabetically
- removed check for not a NULL before calling xfree()
- used unsigned for variables which can't be negative
- removed #include <asm/iommu_fwspec.h> from iommu.h
- added check to iommu_fwspec_init() to not allow overriding
- clarified comments in code
- modified iommu_fwspec_add_ids() to use type-safe xrealloc_flex_struct()

---
 xen/drivers/passthrough/arm/Makefile       |  2 +-
 xen/drivers/passthrough/arm/iommu_fwspec.c | 93 ++++++++++++++++++++++++++++++
 xen/include/asm-arm/device.h               |  1 +
 xen/include/asm-arm/iommu_fwspec.h         | 68 ++++++++++++++++++++++
 4 files changed, 163 insertions(+), 1 deletion(-)
 create mode 100644 xen/drivers/passthrough/arm/iommu_fwspec.c
 create mode 100644 xen/include/asm-arm/iommu_fwspec.h
xen/drivers/passthrough/arm/Makefile +1 -1
index 4abb87a..5fbad45 100644
@@ -1,2 +1,2 @@
-obj-y += iommu.o iommu_helpers.o
+obj-y += iommu.o iommu_helpers.o iommu_fwspec.o
 obj-$(CONFIG_ARM_SMMU) += smmu.o
xen/drivers/passthrough/arm/iommu_fwspec.c new file +93 -0

93 lines not on this page · raw message

xen/include/asm-arm/device.h +1 -0
index ee1c3bc..ee7cff2 100644
@@ -18,6 +18,7 @@ struct device
     struct dt_device_node *of_node; /* Used by drivers imported from Linux */
 #endif
     struct dev_archdata archdata;
+    struct iommu_fwspec *iommu_fwspec; /* per-device IOMMU instance data */
 };
 
 typedef struct device device_t;
xen/include/asm-arm/iommu_fwspec.h new file +68 -0

68 lines not on this page · raw message

Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#20 lore raw

Re: [Xen-devel] [PATCH V4 6/8] iommu/arm: Add lightweight iommu_fwspec support

Quoted 33 lines · Oleksandr Tyshchenko
xen/include/asm-arm/iommu_fwspec.h new file +30 -0
@@ -0,0 +1,68 @@
+/*
+ * xen/include/asm-arm/iommu_fwspec.h
+ *
+ * Contains a common structure to hold the per-device firmware data and
+ * declaration of functions used to maintain that data
+ *
+ * Based on Linux's iommu_fwspec support you can find at:
+ *    include/linux/iommu.h
+ *
+ * Copyright (C) 2007-2008 Advanced Micro Devices, Inc.
+ *
+ * Copyright (C) 2019 EPAM Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __ARCH_ARM_IOMMU_FWSPEC_H__
+#define __ARCH_ARM_IOMMU_FWSPEC_H__
+
+/* per-device IOMMU instance data */

xen/include/asm-arm/iommu_fwspec.h line 40

Quoted 10 lines
xen/include/asm-arm/iommu_fwspec.h new file +10 -0
+struct iommu_fwspec {
+    /* this device's IOMMU */
+    struct device *iommu_dev;
+    /* IOMMU driver private data for this device */
+    void *iommu_priv;
+    /* number of associated device IDs */
+    unsigned int num_ids;
+    /* IDs which this device may present to the IOMMU */
+    uint32_t ids[1];
+};

Note that you abuse xrealloc_flex_struct() when using it with such a type: The last field is _not_ a flexible array member. Compilers might legitimately warn if they can prove that you access p->ids[1] anywhere, despite you (presumably) having allocated enough space. (I haven't been able to think of a way for the macro to actually detect and hence refuse such wrong uses.)

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#21 lore raw

Re: [Xen-devel] [PATCH V4 6/8] iommu/arm: Add lightweight iommu_fwspec support

On 16.09.19 13:40, Jan Beulich wrote:

Hi, Jan

Quoted 11 lines
Quoted 11 lines
xen/include/asm-arm/iommu_fwspec.h new file +10 -0
+
+/* per-device IOMMU instance data */
+struct iommu_fwspec {
+    /* this device's IOMMU */
+    struct device *iommu_dev;
+    /* IOMMU driver private data for this device */
+    void *iommu_priv;
+    /* number of associated device IDs */
+    unsigned int num_ids;
+    /* IDs which this device may present to the IOMMU */
Quoted 8 lines

xen/include/asm-arm/iommu_fwspec.h line 40

Quoted 2 lines
xen/include/asm-arm/iommu_fwspec.h new file +2 -0
+    uint32_t ids[1];
+};

Note that you abuse xrealloc_flex_struct() when using it with such a type: The last field is _not_ a flexible array member. Compilers might legitimately warn if they can prove that you access p->ids[1] anywhere, despite you (presumably) having allocated enough space. (I haven't been able to think of a way for the macro to actually detect and hence refuse such wrong uses.)

Indeed, you are right. I am in doubt, whether to retain ported from Linux code (ids[1])

and mention about such abuse or change it to deal with real flexible array member (ids[]). Any thoughts?

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#22 lore raw

Re: [Xen-devel] [PATCH V4 6/8] iommu/arm: Add lightweight iommu_fwspec support

Quoted 19 lines · Oleksandr
Quoted 9 lines · Jan Beulich
Quoted 9 lines
xen/include/asm-arm/iommu_fwspec.h new file +9 -0
+/* per-device IOMMU instance data */
+struct iommu_fwspec {
+    /* this device's IOMMU */
+    struct device *iommu_dev;
+    /* IOMMU driver private data for this device */
+    void *iommu_priv;
+    /* number of associated device IDs */
+    unsigned int num_ids;
+    /* IDs which this device may present to the IOMMU */
Quoted 8 lines

xen/include/asm-arm/iommu_fwspec.h line 40

Quoted 2 lines
xen/include/asm-arm/iommu_fwspec.h new file +2 -0
+    uint32_t ids[1];
+};

Note that you abuse xrealloc_flex_struct() when using it with such a type: The last field is _not_ a flexible array member. Compilers might legitimately warn if they can prove that you access p->ids[1] anywhere, despite you (presumably) having allocated enough space. (I haven't been able to think of a way for the macro to actually detect and hence refuse such wrong uses.)

Quoted 5 lines

Indeed, you are right. I am in doubt, whether to retain ported from Linux code (ids[1])

and mention about such abuse or change it to deal with real flexible array member (ids[]). Any thoughts?

I'm of the strong opinion that you should switch to [] (or at least [0]) notation.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#23 lore raw

Re: [Xen-devel] [PATCH V4 6/8] iommu/arm: Add lightweight iommu_fwspec support

On 17.09.19 09:12, Jan Beulich wrote:

Hi, Jan

Quoted 19 lines · Oleksandr
Quoted 18 lines · Jan Beulich
Quoted 17 lines

xen/include/asm-arm/iommu_fwspec.h line 40

Quoted 11 lines
xen/include/asm-arm/iommu_fwspec.h new file +11 -0
+/* per-device IOMMU instance data */
+struct iommu_fwspec {
+    /* this device's IOMMU */
+    struct device *iommu_dev;
+    /* IOMMU driver private data for this device */
+    void *iommu_priv;
+    /* number of associated device IDs */
+    unsigned int num_ids;
+    /* IDs which this device may present to the IOMMU */
+    uint32_t ids[1];
+};

Note that you abuse xrealloc_flex_struct() when using it with such a type: The last field is _not_ a flexible array member. Compilers might legitimately warn if they can prove that you access p->ids[1] anywhere, despite you (presumably) having allocated enough space. (I haven't been able to think of a way for the macro to actually detect and hence refuse such wrong uses.)

Quoted 7 lines
Quoted 5 lines

Indeed, you are right. I am in doubt, whether to retain ported from Linux code (ids[1])

and mention about such abuse or change it to deal with real flexible array member (ids[]). Any thoughts?

I'm of the strong opinion that you should switch to [] (or at least [0]) notation.

I got it. Well, will switch to ids[] if there are no objections.

Thank you.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#24 lore raw

Re: [Xen-devel] [PATCH V4 6/8] iommu/arm: Add lightweight iommu_fwspec support

Hi,

Quoted 27 lines · Oleksandr

On 17.09.19 09:12, Jan Beulich wrote:

Hi, Jan

Quoted 22 lines · Oleksandr
Quoted 21 lines · Jan Beulich
Quoted 9 lines
Quoted 9 lines
xen/include/asm-arm/iommu_fwspec.h new file +9 -0
+/* per-device IOMMU instance data */
+struct iommu_fwspec {
+    /* this device's IOMMU */
+    struct device *iommu_dev;
+    /* IOMMU driver private data for this device */
+    void *iommu_priv;
+    /* number of associated device IDs */
+    unsigned int num_ids;
+    /* IDs which this device may present to the IOMMU */
Quoted 8 lines
Quoted 2 lines

+    uint32_t ids[1];
+};

Note that you abuse xrealloc_flex_struct() when using it with such a type: The last field is _not_ a flexible array member. Compilers might legitimately warn if they can prove that you access p->ids[1] anywhere, despite you (presumably) having allocated enough space. (I haven't been able to think of a way for the macro to actually detect and hence refuse such wrong uses.)

Indeed, you are right. I am in doubt, whether to retain ported from Linux code (ids[1])

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

and mention about such abuse or change it to deal with real flexible array member (ids[]). Any thoughts?

I'm of the strong opinion that you should switch to [] (or at least [0]) notation.

I got it. Well, will switch to ids[] if there are no objections.

I suspect the rationale to use 1 rather than 0 is to avoid the re-allocation in the common case where a device has a single ID.

I would like to retain the similar behavior. The ids[1] is probably the most pretty way to do it.

Another solution would to use xmalloc_bytes() for the initial allocation of xmalloc_bytes().

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#25 lore raw

Re: [Xen-devel] [PATCH V4 6/8] iommu/arm: Add lightweight iommu_fwspec support

Quoted 37 lines · Julien Grall

Hi,

Quoted 27 lines · Oleksandr

On 17.09.19 09:12, Jan Beulich wrote:

Hi, Jan

Quoted 22 lines · Oleksandr
Quoted 21 lines · Jan Beulich
Quoted 9 lines
Quoted 9 lines
xen/include/asm-arm/iommu_fwspec.h new file +9 -0
+/* per-device IOMMU instance data */
+struct iommu_fwspec {
+    /* this device's IOMMU */
+    struct device *iommu_dev;
+    /* IOMMU driver private data for this device */
+    void *iommu_priv;
+    /* number of associated device IDs */
+    unsigned int num_ids;
+    /* IDs which this device may present to the IOMMU */
Quoted 8 lines
Quoted 2 lines

+    uint32_t ids[1];
+};

Note that you abuse xrealloc_flex_struct() when using it with such a type: The last field is _not_ a flexible array member. Compilers might legitimately warn if they can prove that you access p->ids[1] anywhere, despite you (presumably) having allocated enough space. (I haven't been able to think of a way for the macro to actually detect and hence refuse such wrong uses.)

Indeed, you are right. I am in doubt, whether to retain ported from Linux code (ids[1])

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

and mention about such abuse or change it to deal with real flexible array member (ids[]). Any thoughts?

I'm of the strong opinion that you should switch to [] (or at least [0]) notation.

I got it. Well, will switch to ids[] if there are no objections.

Quoted 8 lines

I suspect the rationale to use 1 rather than 0 is to avoid the re-allocation in the common case where a device has a single ID.

I would like to retain the similar behavior. The ids[1] is probably the most pretty way to do it.

Another solution would to use xmalloc_bytes() for the initial allocation of xmalloc_bytes().

Why not use xmalloc_flex_<whatever>() on the first allocation, too?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#26 lore raw

Re: [Xen-devel] [PATCH V4 6/8] iommu/arm: Add lightweight iommu_fwspec support

Hi, all.

Quoted 27 lines
Quoted 27 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

Review

Quoted 10 lines
(quoted diff) +10 -0
+struct iommu_fwspec {
+    /* this device's IOMMU */
+    struct device *iommu_dev;
+    /* IOMMU driver private data for this device */
+    void *iommu_priv;
+    /* number of associated device IDs */
+    unsigned int num_ids;
+    /* IDs which this device may present to the IOMMU */
+    uint32_t ids[1];
+};

Note that you abuse xrealloc_flex_struct() when using it with such a type: The last field is _not_ a flexible array member. Compilers might legitimately warn if they can prove that you access p->ids[1] anywhere, despite you (presumably) having allocated enough space. (I haven't been able to think of a way for the macro to actually detect and hence refuse such wrong uses.)

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

Indeed, you are right. I am in doubt, whether to retain ported from Linux code (ids[1])

and mention about such abuse or change it to deal with real flexible array member (ids[]). Any thoughts?

I'm of the strong opinion that you should switch to [] (or at least [0]) notation.

I got it. Well, will switch to ids[] if there are no objections.

I suspect the rationale to use 1 rather than 0 is to avoid the re-allocation in the common case where a device has a single ID.

Quoted 6 lines
Quoted 5 lines

I would like to retain the similar behavior. The ids[1] is probably the most pretty way to do it.

Another solution would to use xmalloc_bytes() for the initial allocation of xmalloc_bytes().

Why not use xmalloc_flex_<whatever>() on the first allocation, too?

Hmm, why not? Looks like the xmalloc_flex_struct fits here.

The first allocation would be:

fwspec = xmalloc_flex_struct(struct iommu_fwspec,  ids, 1);

The re-allocation for the devices with single ID would do effectively nothing (assuming that _xrealloc will recognize that size is the same):

fwspec = xrealloc_flex_struct(fwspec, ids, fwspec->num_ids + num_ids);

Julien, what do you think?

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#27 lore raw

Re: [Xen-devel] [PATCH V4 6/8] iommu/arm: Add lightweight iommu_fwspec support

Quoted 2 lines · Oleksandr

Hi, all.

Hi,

Quoted 42 lines
Quoted 29 lines
Quoted 29 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

Review

Quoted 12 lines
(quoted diff) +10 -0
+struct iommu_fwspec {
+    /* this device's IOMMU */
+    struct device *iommu_dev;
+    /* IOMMU driver private data for this device */
+    void *iommu_priv;
+    /* number of associated device IDs */
+    unsigned int num_ids;
+    /* IDs which this device may present to the IOMMU */
+    uint32_t ids[1];
+};

Note that you abuse xrealloc_flex_struct() when using it with such a type: The last field is _not_ a flexible array member. Compilers might legitimately warn if they can prove that you access p->ids[1] anywhere, despite you (presumably) having allocated enough space. (I haven't been able to think of a way for the macro to actually detect and hence refuse such wrong uses.)

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

Indeed, you are right. I am in doubt, whether to retain ported from Linux code (ids[1])

and mention about such abuse or change it to deal with real flexible array member (ids[]). Any thoughts?

I'm of the strong opinion that you should switch to [] (or at least [0]) notation.

I got it. Well, will switch to ids[] if there are no objections.

I suspect the rationale to use 1 rather than 0 is to avoid the re-allocation in the common case where a device has a single ID.

Quoted 6 lines
Quoted 5 lines

I would like to retain the similar behavior. The ids[1] is probably the most pretty way to do it.

Another solution would to use xmalloc_bytes() for the initial allocation of xmalloc_bytes().

Why not use xmalloc_flex_<whatever>() on the first allocation, too?

Hmm, why not? Looks like the xmalloc_flex_struct fits here.

The first allocation would be:

fwspec = xmalloc_flex_struct(struct iommu_fwspec,  ids, 1);

Quoted 6 lines

The re-allocation for the devices with single ID would do effectively nothing (assuming that _xrealloc will recognize that size is the same):

fwspec = xrealloc_flex_struct(fwspec, ids, fwspec->num_ids + num_ids);

Julien, what do you think?

I am happy with that. The first allocation would need a comment on top explaining the reason of the "1".

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#28 lore raw

Re: [Xen-devel] [PATCH V4 6/8] iommu/arm: Add lightweight iommu_fwspec support

Hi, all.

Quoted 41 lines
Quoted 41 lines
Quoted 28 lines
Quoted 28 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines

Review

Quoted 11 lines
(quoted diff) +10 -0
+struct iommu_fwspec {
+    /* this device's IOMMU */
+    struct device *iommu_dev;
+    /* IOMMU driver private data for this device */
+    void *iommu_priv;
+    /* number of associated device IDs */
+    unsigned int num_ids;
+    /* IDs which this device may present to the IOMMU */
+    uint32_t ids[1];
+};

Note that you abuse xrealloc_flex_struct() when using it with such a type: The last field is _not_ a flexible array member. Compilers might legitimately warn if they can prove that you access p->ids[1] anywhere, despite you (presumably) having allocated enough space. (I haven't been able to think of a way for the macro to actually detect and hence refuse such wrong uses.)

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

Indeed, you are right. I am in doubt, whether to retain ported from Linux code (ids[1])

and mention about such abuse or change it to deal with real flexible array member (ids[]). Any thoughts?

I'm of the strong opinion that you should switch to [] (or at least [0]) notation.

I got it. Well, will switch to ids[] if there are no objections.

I suspect the rationale to use 1 rather than 0 is to avoid the re-allocation in the common case where a device has a single ID.

Quoted 6 lines
Quoted 5 lines

I would like to retain the similar behavior. The ids[1] is probably the most pretty way to do it.

Another solution would to use xmalloc_bytes() for the initial allocation of xmalloc_bytes().

Why not use xmalloc_flex_<whatever>() on the first allocation, too?

Hmm, why not? Looks like the xmalloc_flex_struct fits here.

The first allocation would be:

fwspec = xmalloc_flex_struct(struct iommu_fwspec,  ids, 1);

Quoted 8 lines
Quoted 6 lines

The re-allocation for the devices with single ID would do effectively nothing (assuming that _xrealloc will recognize that size is the same):

fwspec = xrealloc_flex_struct(fwspec, ids, fwspec->num_ids + num_ids);

Julien, what do you think?

I am happy with that. The first allocation would need a comment on top explaining the reason of the "1".

Sure, will add.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#29 lore raw

PATCH v4 7/8 iommu/arm: Introduce iommu_add_dt_device API

In reply to Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

The main puprose of this patch is to add a way to register DT device (which is behind the IOMMU) using the generic IOMMU DT bindings [1] before assigning that device to a domain.

So, this patch adds new "iommu_add_dt_device" API for adding DT device to the IOMMU using generic IOMMU DT bindings and previously added "iommu_fwspec" support. It is called when constructing Dom0 since "iommu_assign_dt_device" can be called for Dom0 also.

Besides that, this patch adds new "dt_xlate" callback (borrowed from Linux "of_xlate") for providing the driver with DT IOMMU specifier which describes the IOMMU master interfaces of that device (device IDs, etc). According to the generic IOMMU DT bindings the context of required properties for IOMMU device/master node (#iommu-cells, iommus) depends on many factors and is really driver depended thing.

Please note, all IOMMU drivers which support generic IOMMU DT bindings should use "dt_xlate" and "add_device" callbacks.

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt

---
Changes V3 -> V4:
- squashed with "iommu: Add of_xlate callback" patch
- renamed "of_xlate" to "dt_xlate"
- reworked patch description
- clarified comments in code, removed confusing word "initialize device", etc
- updated debug message in handle_device()
- modified to check ops->of_xlate and ops->add_device only if "iommus" property is exists

Changes V2 -> V3:
- clarified patch description
- clarified comments in code
- modified to provide DT IOMMU specifier to the driver using "of_xlate" callback
- documented function usage
- modified to return an error if ops is not present/implemented,
- added ability to return a possitive value to indicate that device doesn't need to be protected
- removed check for the "iommu" property presence
in the common code
- included <asm/iommu_fwspec.h> directly

---
 xen/arch/arm/domain_build.c         | 12 +++++++
 xen/drivers/passthrough/arm/iommu.c | 63 +++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/iommu.h         | 11 +++++++
 xen/include/xen/iommu.h             | 10 ++++++
 4 files changed, 96 insertions(+)
xen/arch/arm/domain_build.c +12 -0
index a0fee1e..0d79182 100644
@@ -1240,6 +1240,7 @@ static int __init map_device_children(struct domain *d,
 
 /*
  * For a given device node:
+ *  - Try to call iommu_add_dt_device to protect the device by an IOMMU
  *  - Give permission to the guest to manage IRQ and MMIO range
  *  - Retrieve the IRQ configuration (i.e edge/level) from device tree
  * When the device is not marked for guest passthrough:
@@ -1257,6 +1258,17 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev,
     u64 addr, size;
     bool need_mapping = !dt_device_for_passthrough(dev);
 
+    dt_dprintk("Check if %s is behind the IOMMU and add it\n",
+               dt_node_full_name(dev));
+
+    res = iommu_add_dt_device(dev);
+    if ( res < 0 )
+    {
+        printk(XENLOG_ERR "Failed to add %s to the IOMMU\n",
+               dt_node_full_name(dev));
+        return res;
+    }
+
     nirq = dt_number_of_irq(dev);
     naddr = dt_number_of_address(dev);
 
xen/drivers/passthrough/arm/iommu.c +63 -0

72 lines not on this page · raw message

xen/include/asm-arm/iommu.h +11 -0
index 11dedba..04f21a2 100644
@@ -27,6 +27,17 @@ const struct iommu_ops *iommu_get_ops(void);
 void iommu_set_ops(const struct iommu_ops *ops);
 
 /*
+ * Helper to add master device to the IOMMU using generic IOMMU DT bindings.
+ *
+ * Return values:
+ *  0 : device is protected by an IOMMU
+ * <0 : device is not protected by an IOMMU, but must be (error condition)
+ * >0 : device doesn't need to be protected by an IOMMU
+ *      (IOMMU is not enabled/present or device is not connected to it).
+ */
+int iommu_add_dt_device(struct dt_device_node *np);
+
+/*
  * The mapping helpers below should only be used if P2M Table is shared
  * between the CPU and the IOMMU.
  */
xen/include/xen/iommu.h +10 -0
index ab258b8..59a2cee 100644
@@ -239,6 +239,16 @@ struct iommu_ops {
     int __must_check (*iotlb_flush_all)(struct domain *d);
     int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
     void (*dump_p2m_table)(struct domain *d);
+
+#ifdef CONFIG_HAS_DEVICE_TREE
+    /*
+     * All IOMMU drivers which support generic IOMMU DT bindings should use
+     * this callback. This is a way for the framework to provide the driver
+     * with DT IOMMU specifier which describes the IOMMU master interfaces of
+     * that device (device IDs, etc).
+     */
+    int (*dt_xlate)(device_t *dev, struct dt_phandle_args *args);
+#endif
 };
 
 #include <asm/iommu.h>
Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#30 lore raw

Re: [Xen-devel] [PATCH V4 7/8] iommu/arm: Introduce iommu_add_dt_device API

Quoted 7 lines · Oleksandr Tyshchenko
xen/include/xen/iommu.h +1 -0
@@ -239,6 +239,16 @@ struct iommu_ops {
     int __must_check (*iotlb_flush_all)(struct domain *d);
     int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
     void (*dump_p2m_table)(struct domain *d);
+

xen/include/xen/iommu.h struct iommu_ops { · line 252

Quoted 10 lines
xen/include/xen/iommu.h +9 -0
+#ifdef CONFIG_HAS_DEVICE_TREE
+    /*
+     * All IOMMU drivers which support generic IOMMU DT bindings should use
+     * this callback. This is a way for the framework to provide the driver
+     * with DT IOMMU specifier which describes the IOMMU master interfaces of
+     * that device (device IDs, etc).
+     */
+    int (*dt_xlate)(device_t *dev, struct dt_phandle_args *args);
+#endif
 };

Before I give my ack on this, would you please clarify whether indeed both parameters are intended to be written to by the hook function? If not, either or both should get "const" added.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#31 lore raw

Re: [Xen-devel] [PATCH V4 7/8] iommu/arm: Introduce iommu_add_dt_device API

On 16.09.19 12:53, Jan Beulich wrote:

Hi, Jan

Quoted 13 lines · Oleksandr Tyshchenko
Quoted 12 lines
xen/include/xen/iommu.h +6 -0
@@ -239,6 +239,16 @@ struct iommu_ops {
      int __must_check (*iotlb_flush_all)(struct domain *d);
      int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
      void (*dump_p2m_table)(struct domain *d);
+
+#ifdef CONFIG_HAS_DEVICE_TREE
+    /*
+     * All IOMMU drivers which support generic IOMMU DT bindings should use
+     * this callback. This is a way for the framework to provide the driver
+     * with DT IOMMU specifier which describes the IOMMU master interfaces of
Quoted 8 lines

xen/include/xen/iommu.h struct iommu_ops { · line 252

Quoted 5 lines
xen/include/xen/iommu.h +4 -0
+     * that device (device IDs, etc).
+     */
+    int (*dt_xlate)(device_t *dev, struct dt_phandle_args *args);
+#endif
  };

Before I give my ack on this, would you please clarify whether indeed both parameters are intended to be written to by the hook function? If not, either or both should get "const" added.

Good question. I will add "const" to args parameter.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#32 lore raw

Re: [Xen-devel] [PATCH V4 7/8] iommu/arm: Introduce iommu_add_dt_device API

In reply to Oleksandr Tyshchenko

Hi Oleksandr,

Quoted 10 lines · Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

The main puprose of this patch is to add a way to register DT device (which is behind the IOMMU) using the generic IOMMU DT bindings [1] before assigning that device to a domain.

So, this patch adds new "iommu_add_dt_device" API for adding DT device to the IOMMU using generic IOMMU DT bindings and previously added "iommu_fwspec" support. It is called when constructing Dom0 since "iommu_assign_dt_device" can be called for Dom0 also.

The last sentence is misleading. Yes some devices may be assigned to the hardware domain (aka dom0) but other may be assigned to other domains.

Here you are (ab)using the construction of the hardware domain to add all the devices to the IOMMU. This works fine now because the hardware domain will always be the first domain to be initialized. But I am not sure that whether this is correct to do long term.

As mentioned earlier, my preference is to keep "add" and "assign" separated. So maybe what we want to do is:

if ( need_mapping )
{
iommu_add_dt_device(d, dev);
if ( dt_device_is_protected(d) )

   {
     res = iommu_assign_dt_device(...);
     if ( res )
       /* error */

}
}

We would need similar code in iommu_do_dt_domctl. Although, device should alway be protected in this case.

Quoted 90 lines

Besides that, this patch adds new "dt_xlate" callback (borrowed from Linux "of_xlate") for providing the driver with DT IOMMU specifier which describes the IOMMU master interfaces of that device (device IDs, etc). According to the generic IOMMU DT bindings the context of required properties for IOMMU device/master node (#iommu-cells, iommus) depends on many factors and is really driver depended thing.

Please note, all IOMMU drivers which support generic IOMMU DT bindings should use "dt_xlate" and "add_device" callbacks.

[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt

---
Changes V3 -> V4:
- squashed with "iommu: Add of_xlate callback" patch
- renamed "of_xlate" to "dt_xlate"
- reworked patch description
- clarified comments in code, removed confusing word "initialize device", etc
- updated debug message in handle_device()
- modified to check ops->of_xlate and ops->add_device only if "iommus" property is exists

Changes V2 -> V3:
- clarified patch description
- clarified comments in code
- modified to provide DT IOMMU specifier to the driver using "of_xlate" callback
- documented function usage
- modified to return an error if ops is not present/implemented,
- added ability to return a possitive value to indicate that device doesn't need to be protected
- removed check for the "iommu" property presence
in the common code
- included <asm/iommu_fwspec.h> directly

---
  xen/arch/arm/domain_build.c         | 12 +++++++
  xen/drivers/passthrough/arm/iommu.c | 63 +++++++++++++++++++++++++++++++++++++
  xen/include/asm-arm/iommu.h         | 11 +++++++
  xen/include/xen/iommu.h             | 10 ++++++
  4 files changed, 96 insertions(+)
xen/arch/arm/domain_build.c +12 -0
index a0fee1e..0d79182 100644
@@ -1240,6 +1240,7 @@ static int __init map_device_children(struct domain *d,
 
  /*
   * For a given device node:
+ *  - Try to call iommu_add_dt_device to protect the device by an IOMMU
   *  - Give permission to the guest to manage IRQ and MMIO range
   *  - Retrieve the IRQ configuration (i.e edge/level) from device tree
   * When the device is not marked for guest passthrough:
@@ -1257,6 +1258,17 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev,
      u64 addr, size;
      bool need_mapping = !dt_device_for_passthrough(dev);
 
+    dt_dprintk("Check if %s is behind the IOMMU and add it\n",
+               dt_node_full_name(dev));
+
+    res = iommu_add_dt_device(dev);
+    if ( res < 0 )
+    {
+        printk(XENLOG_ERR "Failed to add %s to the IOMMU\n",
+               dt_node_full_name(dev));
+        return res;
+    }
+
      nirq = dt_number_of_irq(dev);
      naddr = dt_number_of_address(dev);
 
xen/drivers/passthrough/arm/iommu.c +1 -0
index 5a3d1d5..dea79ed 100644
@@ -20,6 +20,7 @@
  #include <xen/lib.h>
 
  #include <asm/device.h>
+#include <asm/iommu_fwspec.h>
 
  /*
   * Deferred probe list is used to keep track of devices for which driver

xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 146

Quoted 5 lines
xen/drivers/passthrough/arm/iommu.c +2 -0
@@ -141,3 +142,65 @@ int arch_iommu_populate_page_table(struct domain *d)
  void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
  {
  }
+
+int __init iommu_add_dt_device(struct dt_device_node *np)

Sorry to only realise it now. Would it make sense to have this function implemented in xen/passthrough/device_tree.c? This would allow to keep arm/iommu.c as firmware agnostic as possible.

Quoted 94 lines
xen/drivers/passthrough/arm/iommu.c +60 -0
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+    struct dt_phandle_args iommu_spec;
+    struct device *dev = dt_to_dev(np);
+    int rc = 1, index = 0;
+
+    if ( !iommu_enabled )
+        return 1;
+
+    if ( !ops )
+        return -EINVAL;
+
+    if ( dev_iommu_fwspec_get(dev) )
+        return -EEXIST;
+
+    /*
+     * According to the Documentation/devicetree/bindings/iommu/iommu.txt
+     * from Linux.
+     */
+    while ( !dt_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+                                        index, &iommu_spec) )
+    {
+        /*
+         * The driver which supports generic IOMMU DT bindings must have
+         * these callback implemented.
+         */
+        if ( !ops->add_device || !ops->dt_xlate )
+            return -EINVAL;
+
+        if ( !dt_device_is_available(iommu_spec.np) )
+            break;
+
+        rc = iommu_fwspec_init(dev, &iommu_spec.np->dev);
+        if ( rc )
+            break;
+
+        /*
+         * Provide DT IOMMU specifier which describes the IOMMU master
+         * interfaces of that device (device IDs, etc) to the driver.
+         * The driver is responsible to decide how to interpret them.
+         */
+        rc = ops->dt_xlate(dev, &iommu_spec);
+        if ( rc )
+            break;
+
+        index++;
+    }
+
+    /*
+     * Add master device to the IOMMU if latter is present and available.
+     * The driver is responsible to mark that device as protected.
+     */
+    if ( !rc )
+        rc = ops->add_device(0, dev);
+
+    if ( rc < 0 )
+        iommu_fwspec_free(dev);
+
+    return rc;
+}
xen/include/asm-arm/iommu.h +11 -0
index 11dedba..04f21a2 100644
@@ -27,6 +27,17 @@ const struct iommu_ops *iommu_get_ops(void);
  void iommu_set_ops(const struct iommu_ops *ops);
 
  /*
+ * Helper to add master device to the IOMMU using generic IOMMU DT bindings.
+ *
+ * Return values:
+ *  0 : device is protected by an IOMMU
+ * <0 : device is not protected by an IOMMU, but must be (error condition)
+ * >0 : device doesn't need to be protected by an IOMMU
+ *      (IOMMU is not enabled/present or device is not connected to it).
+ */
+int iommu_add_dt_device(struct dt_device_node *np);
+
+/*
   * The mapping helpers below should only be used if P2M Table is shared
   * between the CPU and the IOMMU.
   */
xen/include/xen/iommu.h +3 -0
index ab258b8..59a2cee 100644
@@ -239,6 +239,16 @@ struct iommu_ops {
      int __must_check (*iotlb_flush_all)(struct domain *d);
      int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
      void (*dump_p2m_table)(struct domain *d);
+
+#ifdef CONFIG_HAS_DEVICE_TREE
+    /*

xen/include/xen/iommu.h struct iommu_ops { · line 254

Quoted 10 lines
xen/include/xen/iommu.h +7 -0
+     * All IOMMU drivers which support generic IOMMU DT bindings should use
+     * this callback. This is a way for the framework to provide the driver
+     * with DT IOMMU specifier which describes the IOMMU master interfaces of
+     * that device (device IDs, etc).
+     */
+    int (*dt_xlate)(device_t *dev, struct dt_phandle_args *args);
+#endif
  };
 
  #include <asm/iommu.h>

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#33 lore raw

Re: [Xen-devel] [PATCH V4 7/8] iommu/arm: Introduce iommu_add_dt_device API

Quoted 1 line · Julien Grall

Hi Oleksandr,

Hi, Julien.

Quoted 19 lines · Oleksandr Tyshchenko
Quoted 10 lines

From: Oleksandr Tyshchenko <[email protected]>

The main puprose of this patch is to add a way to register DT device (which is behind the IOMMU) using the generic IOMMU DT bindings [1] before assigning that device to a domain.

So, this patch adds new "iommu_add_dt_device" API for adding DT device to the IOMMU using generic IOMMU DT bindings and previously added "iommu_fwspec" support. It is called when constructing Dom0 since "iommu_assign_dt_device" can be called for Dom0 also.

The last sentence is misleading. Yes some devices may be assigned to the hardware domain (aka dom0) but other may be assigned to other domains.

Here you are (ab)using the construction of the hardware domain to add all the devices to the IOMMU. This works fine now because the hardware domain will always be the first domain to be initialized. But I am not sure that whether this is correct to do long term.

As mentioned earlier, my preference is to keep "add" and "assign" separated. So maybe what we want to do is:

Quoted 11 lines

if ( need_mapping )
{
   iommu_add_dt_device(d, dev);
   if ( dt_device_is_protected(d) )
   {
     res = iommu_assign_dt_device(...);
     if ( res )
       /* error */
   }
}
We would need similar code in iommu_do_dt_domctl. Although, device should alway be protected in this case.

Well, will modify this way.

Quoted 11 lines
Quoted 11 lines

  diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c index 5a3d1d5..dea79ed 100644

xen/drivers/passthrough/arm/iommu.c +1 -0
@@ -20,6 +20,7 @@
   #include <xen/lib.h>
     #include <asm/device.h>
+#include <asm/iommu_fwspec.h>
     /*
    * Deferred probe list is used to keep track of devices for which driver
@@ -141,3 +142,65 @@ int arch_iommu_populate_page_table(struct domain 
Quoted 8 lines

xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 146

Quoted 6 lines
xen/drivers/passthrough/arm/iommu.c +0 -0
 *d)
   void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
   {

  }

xen/drivers/passthrough/arm/iommu.c +2 -0
+
+int __init iommu_add_dt_device(struct dt_device_node *np)

Sorry to only realise it now. Would it make sense to have this function implemented in xen/passthrough/device_tree.c?

Not entirely sure. device_tree.c is a common code. The iommu_fwspec stuff (widely used in this function) is ARM code.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#34 lore raw

Re: [Xen-devel] [PATCH V4 7/8] iommu/arm: Introduce iommu_add_dt_device API

Hi,

Quoted 56 lines · Oleksandr
Quoted 19 lines · Julien Grall

Hi Oleksandr,

Quoted 10 lines · Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

The main puprose of this patch is to add a way to register DT device (which is behind the IOMMU) using the generic IOMMU DT bindings [1] before assigning that device to a domain.

So, this patch adds new "iommu_add_dt_device" API for adding DT device to the IOMMU using generic IOMMU DT bindings and previously added "iommu_fwspec" support. It is called when constructing Dom0 since "iommu_assign_dt_device" can be called for Dom0 also.

The last sentence is misleading. Yes some devices may be assigned to the hardware domain (aka dom0) but other may be assigned to other domains.

Here you are (ab)using the construction of the hardware domain to add all the devices to the IOMMU. This works fine now because the hardware domain will always be the first domain to be initialized. But I am not sure that whether this is correct to do long term.

As mentioned earlier, my preference is to keep "add" and "assign" separated. So maybe what we want to do is:

Quoted 11 lines

if ( need_mapping )
{
   iommu_add_dt_device(d, dev);
   if ( dt_device_is_protected(d) )
   {
     res = iommu_assign_dt_device(...);
     if ( res )
       /* error */
   }
}
We would need similar code in iommu_do_dt_domctl. Although, device should alway be protected in this case.

Well, will modify this way.

Quoted 10 lines
Quoted 10 lines

  diff --git a/xen/drivers/passthrough/arm/iommu.c b/xen/drivers/passthrough/arm/iommu.c index 5a3d1d5..dea79ed 100644

xen/drivers/passthrough/arm/iommu.c +1 -0
@@ -20,6 +20,7 @@
   #include <xen/lib.h>
     #include <asm/device.h>
+#include <asm/iommu_fwspec.h>
     /*
    * Deferred probe list is used to keep track of devices for which driver
Quoted 8 lines

xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 146

Quoted 6 lines
xen/drivers/passthrough/arm/iommu.c +2 -0
@@ -141,3 +142,65 @@ int arch_iommu_populate_page_table(struct domain *d)
   void __hwdom_init arch_iommu_hwdom_init(struct domain *d)
   {
   }
+
+int __init iommu_add_dt_device(struct dt_device_node *np)

Sorry to only realise it now. Would it make sense to have this function implemented in xen/passthrough/device_tree.c?

Quoted 1 line

Not entirely sure. device_tree.c is a common code. The iommu_fwspec stuff (widely used in this function) is ARM code.

Some of the device_tree.c already contains Arm specific code (such as device.h).

DT has been only used by Arm so far, so it is sadly fairly tie to the architecture. But it should be easy to make it generic if needs be (such as for RISCv).

While iommu_fwspec is been implemented in Arm headers, this could potentially be made common. So I would still prefer this that function is moved in device_tree.c

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#35 lore raw

Re: [Xen-devel] [PATCH V4 7/8] iommu/arm: Introduce iommu_add_dt_device API

Quoted 1 line · Julien Grall

Hi,

Hi, Julien

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 146

Quoted 3 lines
xen/drivers/passthrough/arm/iommu.c +2 -0
+
+int __init iommu_add_dt_device(struct dt_device_node *np)

Sorry to only realise it now. Would it make sense to have this function implemented in xen/passthrough/device_tree.c?

Quoted 7 lines
Quoted 1 line

Not entirely sure. device_tree.c is a common code. The iommu_fwspec stuff (widely used in this function) is ARM code.

Some of the device_tree.c already contains Arm specific code (such as device.h).

DT has been only used by Arm so far, so it is sadly fairly tie to the architecture. But it should be easy to make it generic if needs be (such as for RISCv).

While iommu_fwspec is been implemented in Arm headers, this could potentially be made common. So I would still prefer this that function is moved in device_tree.c

Well, will move. Also I will remove __init as it can be called at runtime...

As for runtime:

The current implementation allows us to fail at early stage if something is wrong with the device which is behind an IOMMU (and needs to be protected). As we scan for all present devices, but not only for "passthrough".
The "splitting" into handle_device() for hwdom and iommu_do_dt_domctl() for other guests will postpone an error recognition to the guest domain creation time. So, we would have non function system anyway. Wouldn't be better to fail early instead of continue and fail anyway?

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#36 lore raw

Re: [Xen-devel] [PATCH V4 7/8] iommu/arm: Introduce iommu_add_dt_device API

Quoted 5 lines · Oleksandr
Quoted 1 line · Julien Grall

Hi,

Hi, Julien

Hi,

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 146

Quoted 4 lines
xen/drivers/passthrough/arm/iommu.c +2 -0
+
+int __init iommu_add_dt_device(struct dt_device_node *np)

Sorry to only realise it now. Would it make sense to have this function implemented in xen/passthrough/device_tree.c?

Quoted 7 lines
Quoted 1 line

Not entirely sure. device_tree.c is a common code. The iommu_fwspec stuff (widely used in this function) is ARM code.

Some of the device_tree.c already contains Arm specific code (such as device.h).

DT has been only used by Arm so far, so it is sadly fairly tie to the architecture. But it should be easy to make it generic if needs be (such as for RISCv).

While iommu_fwspec is been implemented in Arm headers, this could potentially be made common. So I would still prefer this that function is moved in device_tree.c

Quoted 7 lines

Well, will move. Also I will remove __init as it can be called at runtime...

As for runtime:

The current implementation allows us to fail at early stage if something is wrong with the device which is behind an IOMMU (and needs to be protected). As we scan for all present devices, but not only for "passthrough".
The "splitting" into handle_device() for hwdom and iommu_do_dt_domctl() for other guests will postpone an error recognition to the guest domain creation time. So, we would have non function system anyway. Wouldn't be better to fail early instead of continue and fail anyway?

Yes your implementation allows us to fail at early stage but then you are abusing the function handle_device(). There are actually no promise this will be called for every device going forward. Think about dom0less where the goal is to have no dom0 at all.

You are also tying the order of the domains creation as dom0 would have to be always created before any other domains are created.

Similar (ab)use of handle_device() does not exist, so I would rather not start to introduce them because this will become quickly unmaintainable as we are mixing dom0 creation and Xen initialization.

Even without this series, assigning a device to the guest may not be a success because the IOMMU may not have enough context bank (used for configuring the IOMMU stage-2) or there are not enough memory. Not been able to add the device to the IOMMU driver is only another example where it may fail.

But I would not consider this as not functional. The rest of your system may work perfectly even if this particular device is not usable. There are no security risk as the IOMMU should block any transaction by default.

I am also not in favor of parsing again the Device-Tree (we have enough of them), so this is the best solution I can come up. Feel free to suggest something different.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#37 lore raw

Re: [Xen-devel] [PATCH V4 7/8] iommu/arm: Introduce iommu_add_dt_device API

Hi Julien

Quoted 28 lines
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 146

Quoted 5 lines
xen/drivers/passthrough/arm/iommu.c +2 -0
+
+int __init iommu_add_dt_device(struct dt_device_node *np)

Sorry to only realise it now. Would it make sense to have this function implemented in xen/passthrough/device_tree.c?

Quoted 7 lines
Quoted 1 line

Not entirely sure. device_tree.c is a common code. The iommu_fwspec stuff (widely used in this function) is ARM code.

Some of the device_tree.c already contains Arm specific code (such as device.h).

DT has been only used by Arm so far, so it is sadly fairly tie to the architecture. But it should be easy to make it generic if needs be (such as for RISCv).

While iommu_fwspec is been implemented in Arm headers, this could potentially be made common. So I would still prefer this that function is moved in device_tree.c

Quoted 7 lines

Well, will move. Also I will remove __init as it can be called at runtime...

As for runtime:

The current implementation allows us to fail at early stage if something is wrong with the device which is behind an IOMMU (and needs to be protected). As we scan for all present devices, but not only for "passthrough".
The "splitting" into handle_device() for hwdom and iommu_do_dt_domctl() for other guests will postpone an error recognition to the guest domain creation time. So, we would have non function system anyway. Wouldn't be better to fail early instead of continue and fail anyway?

Yes your implementation allows us to fail at early stage but then you are abusing the function handle_device(). There are actually no promise this will be called for every device going forward. Think about dom0less where the goal is to have no dom0 at all.

You are also tying the order of the domains creation as dom0 would have to be always created before any other domains are created.

Quoted 7 lines

Similar (ab)use of handle_device() does not exist, so I would rather not start to introduce them because this will become quickly unmaintainable as we are mixing dom0 creation and Xen initialization.

Even without this series, assigning a device to the guest may not be a success because the IOMMU may not have enough context bank (used for configuring the IOMMU stage-2) or there are not enough memory. Not been able to add the device to the IOMMU driver is only another example where it may fail.

But I would not consider this as not functional. The rest of your system may work perfectly even if this particular device is not usable. There are no security risk as the IOMMU should block any transaction by default.

I am also not in favor of parsing again the Device-Tree (we have enough of them), so this is the best solution I can come up. Feel free to suggest something different.

I am happy with the explanation, sounds reasonable. Will modify patch as you suggested.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#38 lore raw

PATCH v4 8/8 iommu/arm: Add Renesas IPMMU-VMSA support

In reply to Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.

Please note, current driver is supposed to work only with newest R-Car Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).

The major differences compare to the Linux driver are:

1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.

So, Xen driver is supposed to be used with newest R-Car Gen3 SoC revisions only (H3 ES3.0, M3-W+, etc.) which IPMMU H/W supports stage 2 translation table format.

2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.

3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.

4. IPMMU device tracking. In Xen, all IOMMU devices are managed by single driver instance. So, driver uses global list to keep track of registered IPMMU devices.

CC Julien Grall <[email protected]>
CC Yoshihiro Shimoda <[email protected]>

---
Changes V3 -> V4:
- changed "Gen3" to "R-Car Gen3" in commit description
- renamed "of_xlate" to "dt_xlate"
- removed check for the iommu_hap_pt_share flag in ipmmu_init()
- modified to be able to select IPMMU_VMSA option only if EXPERT is set
- removed an error message when deferring the probe
- moved check for the P2M sharing possibility to the Root IPMMU path

Changes V2 -> V3:
- added Copyright from Linux
- included <xen/iommu.h> and <asm/iommu_fwspec.h> directly
- removed pointless spin_lock from ipmmu_iommu_domain_teardown()
- changed "Gen3" to "R-Car Gen3", "M3 ES3.0" to "M3-W+"
- changed RCAR_PRODUCT_M3 to RCAR_PRODUCT_M3W
- clarified IMTTBCR_TSZ1_MASK
- removed TLB flush from ipmmu_domain_irq() until clarified the purpose
- returned -ENODEV in ipmmu_init() if P2M sharing or P2M IPA size is not supported to show that IOMMU device is not usable in Xen
- implemented new callback "of_xlate" to parse DT IOMMU specifier and add device IDs to "iommu_fwspec"
- reworked "add_device" callback
- updated patch description

Changes V1 -> V2:
- rewrited driver to use iommu_fwspec
- removed DT parsing code for micro-TLBs
- removed struct ipmmu_vmsa_master_cfg, dev_archdata macro
- added ipmmu_find_mmu_by_dev(), various helpers to access fwspec->iommu_priv
- implemented new callback "add_device" to add master device to IPMMU
- removed ipmmu_protect_masters()
- removed code to locate Root device in the first place, used EAGAIN to request deferred probing
- used printk_once for the system wide error messages in ipmmu_init() which don't need to be shown for every device being probed
- removed map_page/unmap_page implementation, reused them from iommu_helpers.c
- used %pd for printing domaid id
- performed various cosmetic fixes
- changed u32 -> uint32_t, u64 -> uint64_t,
unsigned int -> uint32_t where needed
- clarified TODOs
- clafiried supported SoC versions in config IPMMU_VMSA, set default to "n"
- updated comments in code, provided more accurate description, added new comments where needed
- updated patch description by providing differences between Linux/Xen implementations
- removed fields for cache snoop transaction when configuring IMTTBCR (update from Renesas BSP)

---
 xen/arch/arm/platforms/Kconfig           |    1 +
 xen/drivers/passthrough/Kconfig          |   13 +
 xen/drivers/passthrough/arm/Makefile     |    1 +
 xen/drivers/passthrough/arm/ipmmu-vmsa.c | 1332 ++++++++++++++++++++++++++++++
 4 files changed, 1347 insertions(+)
 create mode 100644 xen/drivers/passthrough/arm/ipmmu-vmsa.c
xen/arch/arm/platforms/Kconfig +1 -0
index bc0e9cd..4bb7319 100644
@@ -25,6 +25,7 @@ config RCAR3
 	bool "Renesas RCar3 support"
 	depends on ARM_64
 	select HAS_SCIF
+	select IPMMU_VMSA if EXPERT
 	---help---
 	Enable all the required drivers for Renesas RCar3
 
xen/drivers/passthrough/Kconfig +13 -0
index a3c0649..e22301c 100644
@@ -12,4 +12,17 @@ config ARM_SMMU
 
 	  Say Y here if your SoC includes an IOMMU device implementing the
 	  ARM SMMU architecture.
+
+config IPMMU_VMSA
+	bool "Renesas IPMMU-VMSA found in R-Car Gen3 SoCs" if EXPERT = "y"
+	default n
+	depends on ARM_64
+	---help---
+	  Support for implementations of the Renesas IPMMU-VMSA found
+	  in R-Car Gen3 SoCs.
+
+	  Say Y here if you are using newest R-Car Gen3 SoCs revisions
+	  (H3 ES3.0, M3-W+, etc) which IPMMU hardware supports stage 2
+	  translation table format and is able to use CPU's P2M table as is.
+
 endif
xen/drivers/passthrough/arm/Makefile +1 -0
index 5fbad45..fcd918e 100644
@@ -1,2 +1,3 @@
 obj-y += iommu.o iommu_helpers.o iommu_fwspec.o
 obj-$(CONFIG_ARM_SMMU) += smmu.o
+obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +1332 -0

1,332 lines not on this page · raw message

Signature · 7 lines
2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Acked-by on v4 8/8
#39 lore raw

Re: [Xen-devel] [PATCH V4 8/8] iommu/arm: Add Renesas IPMMU-VMSA support

Hi Oleksandr,

Quoted 30 lines · Oleksandr Tyshchenko

From: Oleksandr Tyshchenko <[email protected]>

The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.

Please note, current driver is supposed to work only with newest R-Car Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).

The major differences compare to the Linux driver are:

1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.

So, Xen driver is supposed to be used with newest R-Car Gen3 SoC revisions only (H3 ES3.0, M3-W+, etc.) which IPMMU H/W supports stage 2 translation table format.

2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.

3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.

Quoted 7 lines

4. IPMMU device tracking. In Xen, all IOMMU devices are managed by single driver instance. So, driver uses global list to keep track of registered IPMMU devices.

CC Julien Grall <[email protected]>
CC Yoshihiro Shimoda <[email protected]>

The Xen specific bits looks good now. Assuming Yoshihiro will give his reviewed-by/acked-by:

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#40 lore raw

Re: [Xen-devel] [PATCH V4 8/8] iommu/arm: Add Renesas IPMMU-VMSA support

Quoted 1 line · Julien Grall

Hi Oleksandr,

Hi Julien

Quoted 36 lines · Oleksandr Tyshchenko
Quoted 34 lines

From: Oleksandr Tyshchenko <[email protected]>

The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.

Please note, current driver is supposed to work only with newest R-Car Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).

The major differences compare to the Linux driver are:

1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.

So, Xen driver is supposed to be used with newest R-Car Gen3 SoC revisions only (H3 ES3.0, M3-W+, etc.) which IPMMU H/W supports stage 2 translation table format.

2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.

3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.

4. IPMMU device tracking. In Xen, all IOMMU devices are managed by single driver instance. So, driver uses global list to keep track of registered IPMMU devices.

Quoted 7 lines
Quoted 3 lines
CC Julien Grall <[email protected]>
CC Yoshihiro Shimoda <[email protected]>

The Xen specific bits looks good now. Assuming Yoshihiro will give his reviewed-by/acked-by:

Thank you!

One note, it turned out [1] that "args" parameter in "dt_xlate" callback needs "const" added (it is not assumed to modify it inside the callback).

This leads to additional changes to the IPMMU driver. If you happy with the changes below, I will retain your A-b.

xen/drivers/passthrough/arm/ipmmu-vmsa.c +0 -0
index b5c18c2..43e4a84 100644
@@ -696,7 +696,7 @@ static void ipmmu_detach_device(struct ipmmu_vmsa_domain *domain,

 }

 static int ipmmu_init_platform_device(struct device *dev,
-                                      struct dt_phandle_args *args)
+                                      const struct dt_phandle_args *args) {
     struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
     struct ipmmu_vmsa_device *mmu;
@@ -1174,7 +1174,8 @@ static int ipmmu_reassign_device(struct domain *s, struct domain *t, return 0;
 }

-static int ipmmu_dt_xlate(struct device *dev, struct dt_phandle_args *spec) +static int ipmmu_dt_xlate(struct device *dev,
+                          const struct dt_phandle_args *spec)
 {
     int ret;

@@ -1187,7 +1188,7 @@ static int ipmmu_dt_xlate(struct device *dev, struct dt_phandle_args *spec) if ( spec->args_count != 1 || spec->args[0] >= IPMMU_UTLB_MAX )
         return -EINVAL;

-    ret = iommu_fwspec_add_ids(dev, spec->args, 1);
+    ret = iommu_fwspec_add_ids(dev, (uint32_t *)spec->args, 1); if ( ret )
         return ret;

[1] https://lists.xenproject.org/archives/html/xen-devel/2019-09/msg01443.html

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#41 lore raw

Re: [Xen-devel] [PATCH V4 8/8] iommu/arm: Add Renesas IPMMU-VMSA support

Hi,

Quoted 86 lines · Oleksandr
Quoted 1 line · Julien Grall

Hi Oleksandr,

Hi Julien

Quoted 36 lines · Oleksandr Tyshchenko
Quoted 34 lines

From: Oleksandr Tyshchenko <[email protected]>

The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.

Please note, current driver is supposed to work only with newest R-Car Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).

The major differences compare to the Linux driver are:

1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.

So, Xen driver is supposed to be used with newest R-Car Gen3 SoC revisions only (H3 ES3.0, M3-W+, etc.) which IPMMU H/W supports stage 2 translation table format.

2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.

3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.

4. IPMMU device tracking. In Xen, all IOMMU devices are managed by single driver instance. So, driver uses global list to keep track of registered IPMMU devices.

Quoted 7 lines
Quoted 3 lines
CC Julien Grall <[email protected]>
CC Yoshihiro Shimoda <[email protected]>

The Xen specific bits looks good now. Assuming Yoshihiro will give his reviewed-by/acked-by:

Thank you!

One note, it turned out [1] that "args" parameter in "dt_xlate" callback needs "const" added (it is not assumed to modify it inside the callback).

This leads to additional changes to the IPMMU driver. If you happy with the changes below, I will retain your A-b.

xen/drivers/passthrough/arm/ipmmu-vmsa.c +1 -1
index b5c18c2..43e4a84 100644
@@ -696,7 +696,7 @@ static void ipmmu_detach_device(struct ipmmu_vmsa_domain 
 *domain,
  }
 
  static int ipmmu_init_platform_device(struct device *dev,
-                                      struct dt_phandle_args *args)
+                                      const struct dt_phandle_args *args)
  {
      struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);

     struct ipmmu_vmsa_device *mmu;

xen/drivers/passthrough/arm/ipmmu-vmsa.c +2 -1
@@ -1174,7 +1174,8 @@ static int ipmmu_reassign_device(struct domain *s, struct 
 domain *t,
      return 0;
  }
 
-static int ipmmu_dt_xlate(struct device *dev, struct dt_phandle_args *spec)
+static int ipmmu_dt_xlate(struct device *dev,
+                          const struct dt_phandle_args *spec)
  {
      int ret;

xen/drivers/passthrough/arm/ipmmu-vmsa.c static int ipmmu_dt_xlate(struct device *dev, struct · line 1192

Quoted 6 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c +1 -1
@@ -1187,7 +1188,7 @@ static int ipmmu_dt_xlate(struct device *dev, struct 
 dt_phandle_args *spec)
      if ( spec->args_count != 1 || spec->args[0] >= IPMMU_UTLB_MAX )
          return -EINVAL;
 
-    ret = iommu_fwspec_add_ids(dev, spec->args, 1);
+    ret = iommu_fwspec_add_ids(dev, (uint32_t *)spec->args, 1);

NAck, you should never use a cast to remove a const. Looking at the code, iommu_fwspec_add_ids() does not need to modify the ids so the const should be propagated.

With that in place, my ack can stand.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#42 lore raw

Re: [Xen-devel] [PATCH V4 8/8] iommu/arm: Add Renesas IPMMU-VMSA support

Quoted 1 line · Julien Grall

Hi,

Hi Julien.

Quoted 94 lines · Oleksandr
Quoted 92 lines · Julien Grall
Quoted 1 line

Hi Oleksandr,

Hi Julien

Quoted 36 lines · Oleksandr Tyshchenko
Quoted 34 lines

From: Oleksandr Tyshchenko <[email protected]>

The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.

Please note, current driver is supposed to work only with newest R-Car Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).

The major differences compare to the Linux driver are:

1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.

So, Xen driver is supposed to be used with newest R-Car Gen3 SoC revisions only (H3 ES3.0, M3-W+, etc.) which IPMMU H/W supports stage 2 translation table format.

2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.

3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.

4. IPMMU device tracking. In Xen, all IOMMU devices are managed by single driver instance. So, driver uses global list to keep track of registered IPMMU devices.

Quoted 7 lines
Quoted 3 lines
CC Julien Grall <[email protected]>
CC Yoshihiro Shimoda <[email protected]>

The Xen specific bits looks good now. Assuming Yoshihiro will give his reviewed-by/acked-by:

Thank you!

One note, it turned out [1] that "args" parameter in "dt_xlate" callback needs "const" added (it is not assumed to modify it inside the callback).

This leads to additional changes to the IPMMU driver. If you happy with the changes below, I will retain your A-b.

xen/drivers/passthrough/arm/ipmmu-vmsa.c +1 -1
index b5c18c2..43e4a84 100644
@@ -696,7 +696,7 @@ static void ipmmu_detach_device(struct 
 ipmmu_vmsa_domain *domain,
   }
 
   static int ipmmu_init_platform_device(struct device *dev,
-                                      struct dt_phandle_args *args)
+                                      const struct dt_phandle_args 
 *args)
   {

      struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); struct ipmmu_vmsa_device *mmu;

xen/drivers/passthrough/arm/ipmmu-vmsa.c +2 -1
@@ -1174,7 +1174,8 @@ static int ipmmu_reassign_device(struct domain 
 *s, struct domain *t,
       return 0;
   }
 
-static int ipmmu_dt_xlate(struct device *dev, struct dt_phandle_args 
 *spec)
+static int ipmmu_dt_xlate(struct device *dev,
+                          const struct dt_phandle_args *spec)
   {

      int ret;

xen/drivers/passthrough/arm/ipmmu-vmsa.c +0 -0
@@ -1187,7 +1188,7 @@ static int ipmmu_dt_xlate(struct device *dev, 
 struct dt_phandle_args *spec)
       if ( spec->args_count != 1 || spec->args[0] >= IPMMU_UTLB_MAX )
           return -EINVAL;
 
Quoted 6 lines

xen/drivers/passthrough/arm/ipmmu-vmsa.c static int ipmmu_dt_xlate(struct device *dev, · line 1192

Quoted 2 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c +1 -1
-    ret = iommu_fwspec_add_ids(dev, spec->args, 1);
+    ret = iommu_fwspec_add_ids(dev, (uint32_t *)spec->args, 1);

NAck, you should never use a cast to remove a const. Looking at the code, iommu_fwspec_add_ids() does not need to modify the ids so the const should be propagated.

With that in place, my ack can stand.

Fair enough, will propagate.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v4 8/8
#43 lore raw

Re: [Xen-devel] [PATCH V4 8/8] iommu/arm: Add Renesas IPMMU-VMSA support

In reply to Oleksandr Tyshchenko

Hi Oleksandr-san,

Quoted 32 lines

From: Oleksandr Tyshchenko, Sent: Saturday, September 14, 2019 12:35 AM

From: Oleksandr Tyshchenko <[email protected]>

The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.

Please note, current driver is supposed to work only with newest R-Car Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).

The major differences compare to the Linux driver are:

1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.

So, Xen driver is supposed to be used with newest R-Car Gen3 SoC revisions only (H3 ES3.0, M3-W+, etc.) which IPMMU H/W supports stage 2 translation table format.

2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.

3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.

Quoted 7 lines

4. IPMMU device tracking. In Xen, all IOMMU devices are managed by single driver instance. So, driver uses global list to keep track of registered IPMMU devices.

CC Julien Grall <[email protected]>
CC Yoshihiro Shimoda <[email protected]>

Thank you for the patch. I have reviewed this patch about the IPMMU hardware bits, so,

Best regard,
Yoshihiro Shimoda

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#44 lore raw

Re: [Xen-devel] [PATCH V4 8/8] iommu/arm: Add Renesas IPMMU-VMSA support

Quoted 1 line · Yoshihiro Shimoda

Hi Oleksandr-san,

Hi, Shimoda-san

Quoted 36 lines
Quoted 36 lines

From: Oleksandr Tyshchenko, Sent: Saturday, September 14, 2019 12:35 AM

From: Oleksandr Tyshchenko <[email protected]>

The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.

Please note, current driver is supposed to work only with newest R-Car Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).

The major differences compare to the Linux driver are:

1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.

So, Xen driver is supposed to be used with newest R-Car Gen3 SoC revisions only (H3 ES3.0, M3-W+, etc.) which IPMMU H/W supports stage 2 translation table format.

2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.

3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.

4. IPMMU device tracking. In Xen, all IOMMU devices are managed by single driver instance. So, driver uses global list to keep track of registered IPMMU devices.

Quoted 7 lines
Quoted 3 lines
CC Julien Grall <[email protected]>
CC Yoshihiro Shimoda <[email protected]>

Thank you for the patch. I have reviewed this patch about the IPMMU hardware bits, so,

Thank you.

...

I have checked new driver version on R-Car Gen3 M3N SoC and can confirm it works.

Signature · 9 lines
Regards,

Oleksandr Tyshchenko


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel