iommu/arm: Add Renesas IPMMU-VMSA support + Linux's iommu_fwspec
v2 cover iommu/arm: Add Renesas IPMMU-VMSA support + Linux's iommu_fwspec
From: Oleksandr Tyshchenko <[email protected]>
The purpose of this patch series is to add IPMMU-VMSA support to Xen on ARM.
Besides new IOMMU driver, this series contains "iommu_fwspec" support and new API iommu_add_dt_device() for adding DT device to IOMMU.
The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.
Please note, this driver is supposed to work only with newest Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).
----------
This driver is based on Linux's IPMMU-VMSA driver from Renesas BSP:
https://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git/tree/drivers/iommu/ipmmu-vmsa.c?h=v4.14.75-ltsi/rcar-3.9.6
and Xen's SMMU driver:xen/drivers/passthrough/arm/smmu.c
Although Xen driver has a lot in common with Linux driver, it is not a "direct ported" copy and should be treated as such.
The major differences compare to the Linux driver are:
1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.
So, Xen driver is supposed to be used with newest Gen3 SoC revisions only (H3 ES3.0, M3 ES3.0, etc.) which IPMMU H/W supports stage 2 translation table format.
2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.
3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.
----------
Driver was tested on Gen3 H3 ES3.0 based boards using current staging (7d1460c xen/arm: optee: fix compilation with GCC 4.8)
in a system with several DMA masters being assigned to different guest domains.
You can find it here:
repo: https://github.com/otyshchenko1/xen.git branch: ipmmu_upstream2
You can find previous discussions here:
[V1] https://lists.xenproject.org/archives/html/xen-devel/2019-06/msg01755.html
Oleksandr Tyshchenko (6):
iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff iommu/arm: Add ability to handle deferred probing request [RFC] xen/common: Introduce _xrealloc function iommu/arm: Add lightweight iommu_fwspec support iommu/arm: Introduce iommu_add_dt_device API
iommu/arm: Add Renesas IPMMU-VMSA support
xen/arch/arm/domain_build.c | 12 + xen/arch/arm/platforms/Kconfig | 1 + xen/common/device_tree.c | 1 + xen/common/xmalloc_tlsf.c | 21 + xen/drivers/passthrough/Kconfig | 13 + xen/drivers/passthrough/arm/Makefile | 3 +- xen/drivers/passthrough/arm/iommu.c | 80 +- xen/drivers/passthrough/arm/iommu_fwspec.c | 91 ++ xen/drivers/passthrough/arm/iommu_helpers.c | 78 ++ xen/drivers/passthrough/arm/ipmmu-vmsa.c | 1342 +++++++++++++++++++++++++++ xen/drivers/passthrough/arm/smmu.c | 48 +- xen/include/asm-arm/device.h | 7 +- xen/include/asm-arm/iommu.h | 12 + xen/include/asm-arm/iommu_fwspec.h | 65 ++ xen/include/xen/device_tree.h | 1 + xen/include/xen/xmalloc.h | 1 + 16 files changed, 1727 insertions(+), 49 deletions(-) create mode 100644 xen/drivers/passthrough/arm/iommu_fwspec.c create mode 100644 xen/drivers/passthrough/arm/iommu_helpers.c create mode 100644 xen/drivers/passthrough/arm/ipmmu-vmsa.c create mode 100644 xen/include/asm-arm/iommu_fwspec.h
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 1/6 iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff
From: Oleksandr Tyshchenko <[email protected]>
Introduce a separate file to keep various helpers which could be used by more than one IOMMU driver in order not to duplicate code.
The first condidates to be moved to the new file are SMMU driver's "map_page/unmap_page" callbacks. There callbacks neither contain any SMMU specific info nor perform any SMMU specific actions and are going to be the same across all IOMMU drivers which H/W IP shares P2M with the CPU like SMMU does.
So, move callbacks to iommu_helpers.c for the upcoming IPMMU driver to be able to re-use them.
--- xen/drivers/passthrough/arm/Makefile | 2 +- xen/drivers/passthrough/arm/iommu_helpers.c | 78 +++++++++++++++++++++++++++++ xen/drivers/passthrough/arm/smmu.c | 48 +----------------- xen/include/asm-arm/iommu.h | 7 +++ 4 files changed, 88 insertions(+), 47 deletions(-) create mode 100644 xen/drivers/passthrough/arm/iommu_helpers.c
xen/drivers/passthrough/arm/Makefile +1 -1
@@ -1,2 +1,2 @@ -obj-y += iommu.o +obj-y += iommu.o iommu_helpers.o obj-$(CONFIG_ARM_SMMU) += smmu.o
xen/drivers/passthrough/arm/iommu_helpers.c new file +78 -0
78 lines not on this page · raw message
xen/drivers/passthrough/arm/smmu.c +2 -46
60 lines not on this page · raw message
xen/include/asm-arm/iommu.h +7 -0
@@ -26,6 +26,13 @@ struct arch_iommuconst struct iommu_ops *iommu_get_ops (void );void iommu_set_ops (const struct iommu_ops *ops); +/* mapping helpers */ +int __must_check arm_iommu_map_page (struct domain *d,dfn_t dfn,mfn_t mfn, +unsigned int flags, +unsigned int *flush_flags); +int __must_check arm_iommu_unmap_page (struct domain *d,dfn_t dfn, +unsigned int *flush_flags); +#endif /* __ARCH_ARM_IOMMU_H__ */ /*
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 1/6] iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff
Hi Oleksandr,
Quoted 6 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
Introduce a separate file to keep various helpers which could be used by more than one IOMMU driver in order not to duplicate code.
The first condidates to be moved to the new file are SMMU driver's
NIT: s/condidates/candidates/
Quoted 31 lines
"map_page/unmap_page" callbacks. There callbacks neither contain any SMMU specific info nor perform any SMMU specific actions and are going to be the same across all IOMMU drivers which H/W IP shares P2M with the CPU like SMMU does.
So, move callbacks to
iommu_helpers.cfor the upcoming IPMMU driver to be able to re-use them.Signed-off-by Oleksandr Tyshchenko <[email protected]>--- xen/drivers/passthrough/arm/Makefile | 2 +- xen/drivers/passthrough/arm/iommu_helpers.c | 78 +++++++++++++++++++++++++++++ xen/drivers/passthrough/arm/smmu.c | 48 +----------------- xen/include/asm-arm/iommu.h | 7 +++ 4 files changed, 88 insertions(+), 47 deletions(-)create mode 100644
xen/drivers/passthrough/arm/iommu_helpers.cxen/drivers/passthrough/arm/Makefile +1
-1@@ -1,2 +1,2 @@ -obj-y += iommu.o +obj-y += iommu.o iommu_helpers.o obj-$(CONFIG_ARM_SMMU) += smmu.oxen/drivers/passthrough/arm/iommu_helpers.c new file +0
-0
xen/drivers/passthrough/arm/iommu_helpers.c line 6
Quoted 6 lines
xen/drivers/passthrough/arm/iommu_helpers.c new file +6
-0@@ -0,0 +1,78 @@ +/* +* xen/drivers/passthrough/arm/iommu_helpers.c +* +* Contains various helpers to be used by IOMMU drivers. +* +* Copyright (C) 2019 EPAM Systems Inc.
You mostly moved the code from the SMMU code, so the copyright there should be retain. As this is Xen modification the copyright here should be:
* Copyright (C) 2014 Linaro Limited.
Quoted 7 lines
xen/drivers/passthrough/arm/iommu_helpers.c new file +7
-0+* +* This program is free software; you can redistribute it and/or +* modify it under the terms and conditions of the GNU General Public +* License, version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of
xen/drivers/passthrough/arm/iommu_helpers.c line 23
Quoted 10 lines
xen/drivers/passthrough/arm/iommu_helpers.c new file +10
-0+* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public +* License along with this program; If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/lib.h> +#include <xen/sched.h> +#include <xen/iommu.h>
Could you order the headers above alphabetical please?
And also as a extra NIT a newline before 'xen' headers and 'asm' one :).
xen/drivers/passthrough/arm/iommu_helpers.c line 24
Quoted 1 line
xen/drivers/passthrough/arm/iommu_helpers.c new file +1
-0+#include <asm/device.h>
The rest of the good looks good to me.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 1/6] iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff
Hi, Julien
Quoted 10 lines · Oleksandr Tyshchenko
Quoted 6 lines
From: Oleksandr Tyshchenko <[email protected]>
Introduce a separate file to keep various helpers which could be used by more than one IOMMU driver in order not to duplicate code.
The first condidates to be moved to the new file are SMMU driver's
NIT: s/condidates/candidates/
ok
Quoted 42 lines
Quoted 33 lines
"map_page/unmap_page" callbacks. There callbacks neither contain any SMMU specific info nor perform any SMMU specific actions and are going to be the same across all IOMMU drivers which H/W IP shares P2M with the CPU like SMMU does.
So, move callbacks to
iommu_helpers.cfor the upcoming IPMMU driver to be able to re-use them.Signed-off-by Oleksandr Tyshchenko <[email protected]>---
xen/drivers/passthrough/arm/Makefile | 2 +-xen/drivers/passthrough/arm/iommu_helpers.c| 78
+++++++++++++++++++++++++++++
xen/drivers/passthrough/arm/smmu.c | 48 +----------------- xen/include/asm-arm/iommu.h | 7 +++4 files changed, 88 insertions(+), 47 deletions(-)create mode 100644
xen/drivers/passthrough/arm/iommu_helpers.cxen/drivers/passthrough/arm/Makefile +1
-1@@ -1,2 +1,2 @@ -obj-y += iommu.o +obj-y += iommu.o iommu_helpers.oobj-$(CONFIG_ARM_SMMU) += smmu.o
xen/drivers/passthrough/arm/iommu_helpers.c new file +0
-0xen/drivers/passthrough/arm/iommu_helpers.c line 6
Quoted 6 lines
xen/drivers/passthrough/arm/iommu_helpers.c new file +6
-0@@ -0,0 +1,78 @@ +/* +* xen/drivers/passthrough/arm/iommu_helpers.c +* +* Contains various helpers to be used by IOMMU drivers. +* +* Copyright (C) 2019 EPAM Systems Inc. You mostly moved the code from the SMMU code, so the copyright there should be retain. As this is Xen modification the copyright here should be:
Quoted 1 line
* Copyright (C) 2014 Linaro Limited.
Oh, yes. Sorry, forgot about it. Will add.
Quoted 22 lines
Quoted 15 lines
xen/drivers/passthrough/arm/iommu_helpers.c new file +12
-0+* +* This program is free software; you can redistribute it and/or +* modify it under the terms and conditions of the GNU General Public +* License, version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public +* License along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */xen/drivers/passthrough/arm/iommu_helpers.c line 23
Quoted 4 lines
xen/drivers/passthrough/arm/iommu_helpers.c new file +4
-0+ +#include <xen/lib.h> +#include <xen/sched.h> +#include <xen/iommu.h> Could you order the headers above alphabetical please?
Quoted 1 line
And also as a extra NIT a newline before 'xen' headers and 'asm' one :).
Yes, will follow this rule.
Quoted 7 lines
Quoted 2 lines
+#include <
asm/device.h>The rest of the good looks good to me.
Cheers,
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 2/6 iommu/arm: Add ability to handle deferred probing request
In reply to Oleksandr TyshchenkoFrom: Oleksandr Tyshchenko <[email protected]>
This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.
In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen -EAGAIN to be used for indicating that device probing is deferred.
This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.
--- xen/common/device_tree.c | 1 + xen/drivers/passthrough/arm/iommu.c | 35 ++++++++++++++++++++++++++++++++++- xen/include/asm-arm/device.h | 6 +++++- xen/include/xen/device_tree.h | 1 + 4 files changed, 41 insertions(+), 2 deletions(-)
xen/common/device_tree.c +1 -0
@@ -1774,6 +1774,7 @@ static unsigned long __init unflatten_dt_node(const void *fdt,/* By default the device is not protected */ np->is_protected = false;INIT_LIST_HEAD (&np->domain_list); +INIT_LIST_HEAD (&np->deferred_probe);if ( new_format ) {
xen/drivers/passthrough/arm/iommu.c +34 -1
@@ -20,6 +20,12 @@#include <xen/device_tree.h> #include <asm/device.h> +/* +* Used to keep track of devices for which driver requested deferred probing +* (returns -EAGAIN). +*/ +static LIST_HEAD (deferred_probe_list); +static const struct iommu_ops *iommu_ops;const struct iommu_ops *iommu_get_ops (void )
@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops)int __init iommu_hardware_setup (void ) { -struct dt_device_node *np; +struct dt_device_node *np, *tmp;int rc;unsigned int num_iommus =0 ;
@@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void) rc =device_init (np,DEVICE_IOMMU ,NULL );if ( !rc ) num_iommus++; +else if (rc == -EAGAIN ) +/* +* Driver requested deferred probing, so add this device to +* the deferred list for further processing. +*/ +list_add (&np->deferred_probe, &deferred_probe_list); + } + +/* +* Process devices in the deferred list if at least one successfully +* probed device is present. +*/ +while ( !list_empty (&deferred_probe_list) && num_iommus ) + { +list_for_each_entry_safe ( np, tmp, &deferred_probe_list, + deferred_probe ) + { + rc =device_init (np,DEVICE_IOMMU ,NULL ); +if ( !rc ) + num_iommus++; +if ( rc != -EAGAIN ) +/* +* Driver didn't request deferred probing, so remove this device +* from the deferred list. +*/ +list_del_init (&np->deferred_probe); + } }return ( num_iommus >0 ) ?0 : -ENODEV ;
xen/include/asm-arm/device.h +5 -1
@@ -44,7 +44,11 @@ struct device_desc {enum device_class class;/* List of devices supported by this driver */ const struct dt_device_match *dt_match; -/* Device initialization */ +/* +* Device initialization. +* +* -EAGAIN is used to indicate that device probing is deferred. +*/ int (*init)(struct dt_device_node *dev,const void *data); };
xen/include/xen/device_tree.h +1 -0
@@ -93,6 +93,7 @@ struct dt_device_node {/* IOMMU specific fields */ bool is_protected;struct list_head domain_list; +struct list_head deferred_probe;struct device dev; };
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request
Hi Oleksandr,
Quoted 32 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.
In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen
-EAGAINto be used for indicating that device probing is deferred.This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.
Signed-off-by Oleksandr Tyshchenko <[email protected]>--- xen/common/device_tree.c | 1 + xen/drivers/passthrough/arm/iommu.c | 35 ++++++++++++++++++++++++++++++++++- xen/include/asm-arm/device.h | 6 +++++- xen/include/xen/device_tree.h | 1 + 4 files changed, 41 insertions(+), 2 deletions(-)xen/common/device_tree.c +0
-0
xen/common/device_tree.c static unsigned long __init unflatten_dt_node(const void *fdt, · line 1777
Quoted 4 lines
xen/common/device_tree.c +1
-0@@ -1774,6 +1774,7 @@ static unsigned long __init unflatten_dt_node(const void *fdt,/* By default the device is not protected */ np->is_protected = false;INIT_LIST_HEAD (&np->domain_list); +INIT_LIST_HEAD (&np->deferred_probe);
I am not entirely happy to add a new list_head field per node just for the benefits of boot code. Could we re-use domain_list (with a comment in the code and appropriate ASSERT)?
Quoted 8 lines
xen/common/device_tree.c +0
-0if ( new_format ) {xen/drivers/passthrough/arm/iommu.c +0
-0
xen/drivers/passthrough/arm/iommu.c line 27
Quoted 8 lines
xen/drivers/passthrough/arm/iommu.c +5
-0@@ -20,6 +20,12 @@#include <xen/device_tree.h> #include <asm/device.h> +/* +* Used to keep track of devices for which driver requested deferred probing +* (returns -EAGAIN). +*/ +static LIST_HEAD (deferred_probe_list);
This wants to be in init section as this is only used at boot.
Quoted 19 lines
xen/drivers/passthrough/arm/iommu.c +4
-1+static const struct iommu_ops *iommu_ops;const struct iommu_ops *iommu_get_ops (void )@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops)int __init iommu_hardware_setup (void ) { -struct dt_device_node *np; +struct dt_device_node *np, *tmp;int rc;unsigned int num_iommus =0 ;@@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void) rc =device_init (np,DEVICE_IOMMU ,NULL );if ( !rc ) num_iommus++; +else if (rc == -EAGAIN ) +/*
xen/drivers/passthrough/arm/iommu.c int __init iommu_hardware_setup(void) · line 71
Quoted 10 lines
xen/drivers/passthrough/arm/iommu.c +10
-0+* Driver requested deferred probing, so add this device to +* the deferred list for further processing. +*/ +list_add (&np->deferred_probe, &deferred_probe_list); + } + +/* +* Process devices in the deferred list if at least one successfully +* probed device is present. +*/
I think this can turn into an infinite loop if all device in deferred_probe_list still return -EDEFER_PROBE and num_iommus is a non-zero.
A better condition would be to check that at least one IOMMU is added at each loop. If not, then we should bail with an error because it likely means something is buggy.
Quoted 41 lines
xen/drivers/passthrough/arm/iommu.c +15
-0+while ( !list_empty (&deferred_probe_list) && num_iommus ) + { +list_for_each_entry_safe ( np, tmp, &deferred_probe_list, + deferred_probe ) + { + rc =device_init (np,DEVICE_IOMMU ,NULL ); +if ( !rc ) + num_iommus++; +if ( rc != -EAGAIN ) +/* +* Driver didn't request deferred probing, so remove this device +* from the deferred list. +*/ +list_del_init (&np->deferred_probe); + } }return ( num_iommus >0 ) ?0 : -ENODEV ;xen/include/asm-arm/device.h +5
-1@@ -44,7 +44,11 @@ struct device_desc {enum device_class class;/* List of devices supported by this driver */ const struct dt_device_match *dt_match; -/* Device initialization */ +/* +* Device initialization. +* +* -EAGAIN is used to indicate that device probing is deferred. +*/ int (*init)(struct dt_device_node *dev,const void *data); };xen/include/xen/device_tree.h +0
-0
xen/include/xen/device_tree.h struct dt_device_node { · line 99
Quoted 7 lines
xen/include/xen/device_tree.h +1
-0@@ -93,6 +93,7 @@ struct dt_device_node {/* IOMMU specific fields */ bool is_protected;struct list_head domain_list; +struct list_head deferred_probe;struct device dev; };
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi, Julien
Quoted 30 lines · Oleksandr Tyshchenko
Quoted 28 lines
From: Oleksandr Tyshchenko <[email protected]>
This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.
In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen
-EAGAINto be used for indicating that device probing is deferred.This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.
Signed-off-by Oleksandr Tyshchenko <[email protected]>---
xen/common/device_tree.c | 1 +
xen/drivers/passthrough/arm/iommu.c| 35
++++++++++++++++++++++++++++++++++-
xen/include/asm-arm/device.h | 6 +++++-
xen/include/xen/device_tree.h | 1 +4 files changed, 41 insertions(+), 2 deletions(-)
Quoted 12 lines
xen/common/device_tree.c static unsigned long __init · line 1778
Quoted 10 lines
xen/common/device_tree.c +1
-0@@ -1774,6 +1774,7 @@ static unsigned long __init unflatten_dt_node(const void *fdt, /* By default the device is not protected */ np->is_protected = false; INIT_LIST_HEAD(&np->domain_list); + INIT_LIST_HEAD(&np->deferred_probe);I am not entirely happy to add a new
list_headfield per node just for the benefits of boot code. Could we re-usedomain_list(with a comment in the code and appropriate ASSERT)?
Agree that only boot code uses deferred_probe field. I will consider re-using domain_list. Could you please clarify regarding ASSERT (where to put and what to check).
Quoted 10 lines
Quoted 10 lines
xen/common/device_tree.c +0
-0if ( new_format ) {xen/drivers/passthrough/arm/iommu.c +0
-0@@ -20,6 +20,12 @@ #include <xen/device_tree.h> #include <asm/device.h>
Quoted 8 lines
Quoted 6 lines
+/*
+ * Used to keep track of devices for which driver requested deferred probing
+ * (returns-EAGAIN).
+ */
+staticLIST_HEAD(deferred_probe_list);This wants to be in init section as this is only used at boot.
Will do.
Quoted 23 lines
Quoted 23 lines
+
static const structiommu_ops*iommu_ops;
const structiommu_ops*iommu_get_ops(void)xen/drivers/passthrough/arm/iommu.c +1
-1@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops) int __init iommu_hardware_setup(void) { - struct dt_device_node *np; + struct dt_device_node *np, *tmp; int rc; unsigned int num_iommus = 0; @@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void) rc = device_init(np, DEVICE_IOMMU, NULL);if ( !rc )
num_iommus++;
+ else if (rc ==-EAGAIN)
+ /*
+ * Driver requested deferred probing, so add this device to + * the deferred list for further processing.
+ */
+list_add(&np->deferred_probe, &deferred_probe_list); + }
Quoted 8 lines
Quoted 6 lines
+
+ /*
+ * Process devices in the deferred list if at least one successfully
+ * probed device is present.
+ */I think this can turn into an infinite loop if all device in
deferred_probe_liststill return -EDEFER_PROBE andnum_iommusis a non-zero.
Agree.
Quoted 2 lines
A better condition would be to check that at least one IOMMU is added at each loop. If not, then we should bail with an error because it likely means something is buggy.
Sounds reasonable. Will do.
Just to clarify:
Quoted 1 line
Quoted 1 line
Quoted 1 line
A better condition would be to check that at least one IOMMU is added at each loop.
Maybe, not only added (rc == 0), but driver didn't request deferred probe (rc != -EAGAIN).
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request
Hi Oleksandr,
Quoted 43 lines · Oleksandr
Quoted 29 lines · Julien Grall
Quoted 28 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.
In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen
-EAGAINto be used for indicating that device probing is deferred.This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.
Signed-off-by Oleksandr Tyshchenko <[email protected]>---
xen/common/device_tree.c | 1 +
xen/drivers/passthrough/arm/iommu.c| 35
++++++++++++++++++++++++++++++++++-
xen/include/asm-arm/device.h | 6 +++++-
xen/include/xen/device_tree.h | 1 +4 files changed, 41 insertions(+), 2 deletions(-)Quoted 12 lines
xen/common/device_tree.c static unsigned long __init · line 1778
Quoted 10 lines
xen/common/device_tree.c +1
-0@@ -1774,6 +1774,7 @@ static unsigned long __init unflatten_dt_node(const void *fdt, /* By default the device is not protected */ np->is_protected = false; INIT_LIST_HEAD(&np->domain_list); + INIT_LIST_HEAD(&np->deferred_probe);I am not entirely happy to add a new
list_headfield per node just for the benefits of boot code. Could we re-usedomain_list(with a comment in the code and appropriate ASSERT)?
Quoted 1 line
Agree that only boot code uses
deferred_probefield. I will consider re-usingdomain_list. Could you please clarify regarding ASSERT (where to put and what to check).
What I meant is adding an ASSERT to check that np->domain_list is at empty at least before trying to add in the list. This would help to debug any potential issue if we end up to use domain_list earlier in the future. I can't see why it would as iommu is called earlier, but who knows :).
Quoted 65 lines
Quoted 12 lines
Quoted 12 lines
xen/common/device_tree.c +0
-0if ( new_format ) {xen/drivers/passthrough/arm/iommu.c +0
-0@@ -20,6 +20,12 @@ #include <xen/device_tree.h> #include <asm/device.h>Quoted 8 lines
Quoted 6 lines
+/*
+ * Used to keep track of devices for which driver requested deferred probing
+ * (returns-EAGAIN).
+ */
+staticLIST_HEAD(deferred_probe_list);This wants to be in init section as this is only used at boot.
Will do.
Quoted 23 lines
Quoted 23 lines
+
static const structiommu_ops*iommu_ops;
const structiommu_ops*iommu_get_ops(void)xen/drivers/passthrough/arm/iommu.c +1
-1@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops) int __init iommu_hardware_setup(void) { - struct dt_device_node *np; + struct dt_device_node *np, *tmp; int rc; unsigned int num_iommus = 0; @@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void) rc = device_init(np, DEVICE_IOMMU, NULL);if ( !rc )
num_iommus++;
+ else if (rc ==-EAGAIN)
+ /*
+ * Driver requested deferred probing, so add this device to + * the deferred list for further processing.
+ */
+list_add(&np->deferred_probe, &deferred_probe_list); + }Quoted 8 lines
Quoted 6 lines
+
+ /*
+ * Process devices in the deferred list if at least one successfully
+ * probed device is present.
+ */I think this can turn into an infinite loop if all device in
deferred_probe_liststill return -EDEFER_PROBE andnum_iommusis a non-zero.Agree.
Quoted 2 lines
A better condition would be to check that at least one IOMMU is added at each loop. If not, then we should bail with an error because it likely means something is buggy.
Sounds reasonable. Will do.
Quoted 6 lines
Just to clarify:
Quoted 1 line
Quoted 1 line
Quoted 1 line
A better condition would be to check that at least one IOMMU is
added at each loop.
Maybe, not only added (rc == 0), but driver didn't request deferred probe (rc !=
-EAGAIN).
I think adding an IOMMU is enough. If you return an error other than -EAGAIN here after deferring probing, then you are likely going to fail at the next loop. So better to stop early.
I realize this not what the current code is doing (I know I wrote it ;)). But I am not sure it is sane to continue if only part of the IOMMUs are initialized. Most likely you will see an error much later that may be not trivial to find out.
Imagine you want to passthrough you network card to a guest but the IOMMU initialization failed...
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi, Julien
Quoted 45 lines · Oleksandr
Quoted 43 lines · Julien Grall
Quoted 41 lines · Oleksandr Tyshchenko
Quoted 33 lines
From: Oleksandr Tyshchenko <[email protected]>
This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.
In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen
-EAGAINto be used for indicating that device probing is deferred.This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.
Signed-off-by Oleksandr Tyshchenko <[email protected]>---
xen/common/device_tree.c | 1 +
xen/drivers/passthrough/arm/iommu.c| 35
++++++++++++++++++++++++++++++++++-
xen/include/asm-arm/device.h | 6 +++++-
xen/include/xen/device_tree.h | 1 +4 files changed, 41 insertions(+), 2 deletions(-)xen/common/device_tree.c +0
-0xen/common/device_tree.c static unsigned long __init · line 1778
Quoted 5 lines
xen/common/device_tree.c +1
-0@@ -1774,6 +1774,7 @@ static unsigned long __init unflatten_dt_node(const void *fdt, /* By default the device is not protected */ np->is_protected = false; INIT_LIST_HEAD(&np->domain_list); + INIT_LIST_HEAD(&np->deferred_probe);I am not entirely happy to add a new
list_headfield per node just for the benefits of boot code. Could we re-usedomain_list(with a comment in the code and appropriate ASSERT)?
Quoted 3 lines
Quoted 1 line
Agree that only boot code uses
deferred_probefield. I will consider re-usingdomain_list. Could you please clarify regarding ASSERT (where to put and what to check).What I meant is adding an ASSERT to check that
np->domain_listis at empty at least before trying to add in the list. This would help to debug any potential issue if we end up to usedomain_listearlier in the future. I can't see why it would as iommu is called earlier, but who knows :).
Got it. Thank you for clarification.
Quoted 43 lines
Quoted 43 lines
Quoted 23 lines
Quoted 23 lines
xen/common/device_tree.c +1
-0+ static const struct iommu_ops *iommu_ops;const struct
iommu_ops*iommu_get_ops(void)xen/common/device_tree.c +1
-1@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops) int __init iommu_hardware_setup(void) { - struct dt_device_node *np; + struct dt_device_node *np, *tmp; int rc; unsigned int num_iommus = 0; @@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void)rc =
device_init(np, DEVICE_IOMMU, NULL);
if ( !rc )
num_iommus++;
+ else if (rc ==-EAGAIN)
+ /*
+ * Driver requested deferred probing, so add this device to
+ * the deferred list for further processing. + */
+list_add(&np->deferred_probe, &deferred_probe_list); + }Quoted 8 lines
Quoted 6 lines
+
+ /*
+ * Process devices in the deferred list if at least one successfully
+ * probed device is present.
+ */I think this can turn into an infinite loop if all device in
deferred_probe_liststill return -EDEFER_PROBE andnum_iommusis a non-zero.Agree.
Quoted 2 lines
A better condition would be to check that at least one IOMMU is added at each loop. If not, then we should bail with an error because it likely means something is buggy.
Sounds reasonable. Will do.
Just to clarify:
Quoted 7 lines
Quoted 3 lines
>>> A better condition would be to check that at least one IOMMU is added at each loop.
Maybe, not only added (rc == 0), but driver didn't request deferred probe (rc !=
-EAGAIN).I think adding an IOMMU is enough. If you return an error other than
-EAGAINhere after deferring probing, then you are likely going to fail at the next loop. So better to stop early.
It makes sense.
Quoted 5 lines
I realize this not what the current code is doing (I know I wrote it ;)). But I am not sure it is sane to continue if only part of the IOMMUs are initialized. Most likely you will see an error much later that may be not trivial to find out.
Imagine you want to passthrough you network card to a guest but the IOMMU initialization failed...
Oh, agree.
As I understand, the new strict logic would be the following:
If initialization for at least one IOMMU device failed (device_init returns an error other than -EAGAIN), we should stop and return an error to upper layer (even if num_iommus > 0). No matter whether it is during the first attempt or after deferring probe. We don't allow the "I/O virtualisation" to be enabled (iommu_enabled == true) with only part of the IOMMU devices being initialized. Is my understanding correct?
Quoted 5 lines
Cheers,
Signature · 1 line
Julien Grall
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request
Hi Oleksandr,
Quoted 111 lines · Oleksandr
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi, Julien
Quoted 44 lines · Oleksandr
Quoted 42 lines · Julien Grall
Quoted 40 lines · Oleksandr Tyshchenko
Quoted 32 lines
From: Oleksandr Tyshchenko <[email protected]>
This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.
In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen
-EAGAINto be used for indicating that device probing is deferred.This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.
Signed-off-by Oleksandr Tyshchenko <[email protected]>---
xen/common/device_tree.c | 1 +xen/drivers/passthrough/arm/iommu.c| 35 ++++++++++++++++++++++++++++++++++- xen/include/asm-arm/device.h | 6 +++++- xen/include/xen/device_tree.h | 1 +4 files changed, 41 insertions(+), 2 deletions(-)xen/common/device_tree.c +0
-0xen/common/device_tree.c static unsigned long __init unflatten_dt_node(const · line 1778
Quoted 5 lines
xen/common/device_tree.c +1
-0@@ -1774,6 +1774,7 @@ static unsigned long __init unflatten_dt_node(constvoid *fdt, /* By default the device is not protected */ np->is_protected = false; INIT_LIST_HEAD(&np->domain_list); + INIT_LIST_HEAD(&np->deferred_probe);I am not entirely happy to add a new
list_headfield per node just for the benefits of boot code. Could we re-usedomain_list(with a comment in the code and appropriate ASSERT)?Quoted 3 lines
Quoted 1 line
Agree that only boot code uses
deferred_probefield. I will consider re-usingdomain_list. Could you please clarify regarding ASSERT (where to put and what to check).What I meant is adding an ASSERT to check that
np->domain_listis at empty at least before trying to add in the list. This would help to debug any potential issue if we end up to usedomain_listearlier in the future. I can't see why it would as iommu is called earlier, but who knows :).Got it. Thank you for clarification.
Quoted 38 lines
Quoted 38 lines
Quoted 20 lines
Quoted 20 lines
xen/common/device_tree.c +1
-0+ static const struct iommu_ops *iommu_ops;const struct
iommu_ops*iommu_get_ops(void)xen/common/device_tree.c +1
-1@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops) int __init iommu_hardware_setup(void) { - struct dt_device_node *np; + struct dt_device_node *np, *tmp; int rc; unsigned int num_iommus = 0; @@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void) rc = device_init(np, DEVICE_IOMMU, NULL);if ( !rc )
num_iommus++;
+ else if (rc ==-EAGAIN)
+ /*
+ * Driver requested deferred probing, so add this device to + * the deferred list for further processing.
+ */
+list_add(&np->deferred_probe, &deferred_probe_list);Quoted 8 lines
Quoted 6 lines
+ }
+
+ /*
+ * Process devices in the deferred list if at least one successfully + * probed device is present.
+ */I think this can turn into an infinite loop if all device in
deferred_probe_liststill return -EDEFER_PROBE andnum_iommusis a non-zero.Agree.
Quoted 2 lines
A better condition would be to check that at least one IOMMU is added at each loop. If not, then we should bail with an error because it likely means something is buggy.
Sounds reasonable. Will do.
Quoted 7 lines
Quoted 5 lines
Just to clarify:
>>> A better condition would be to check that at least one IOMMU is added at each loop.
Maybe, not only added (rc == 0), but driver didn't request deferred probe (rc !=
-EAGAIN).I think adding an IOMMU is enough. If you return an error other than
-EAGAINhere after deferring probing, then you are likely going to fail at the next loop. So better to stop early.It makes sense.
Quoted 4 lines
I realize this not what the current code is doing (I know I wrote it ;)). But I am not sure it is sane to continue if only part of the IOMMUs are initialized. Most likely you will see an error much later that may be not trivial to find out.
Quoted 7 lines
Quoted 1 line
Imagine you want to passthrough you network card to a guest but the IOMMU initialization failed...
Oh, agree.
As I understand, the new strict logic would be the following:
If initialization for at least one IOMMU device failed (
device_initreturns an error other than-EAGAIN), we should stop and return an error to upper layer (even ifnum_iommus> 0). No matter whether it is during the first attempt or after deferring probe. We don't allow the "I/O virtualisation" to be enabled (iommu_enabled== true) with only part of the IOMMU devices being initialized. Is my understanding correct?
Let me summarize the discussion we had on IRC :). Without your patch, Xen may initialize only half the IOMMUs. If the device is behind an IOMMU that wasn't initialized, then we have two possibility:
1) The device was already mark as protected (if using the old binding in the
SMMU). Xen will not be able to assign the device to Dom0 and therefore Xen will crash (not able to build dom0). For domU, it will depend whether the configuration contain the options 'dtdev'. If the option is specified, then guest will fail to build. On the contrary if the option isn't specified then the guest will boot and this could either lead to transaction failure (if the IOMMU was already reset) or bypassing the IOMMU. Note that the latter can today happen if your IOMMU was disabled. But we can't do much against it.
2) The device is not marked as protected. Xen will not be able to "assign"
the device to Dom0 and this could either lead to the device bypassing the IOMMU or a transaction failure. For domU, the problem is similar to 1).
In the case of the SMMU driver, we only support old bindings. So devices are marked as protected during SMMU initialization. This is done before the SMMU is reset. Before reset the SMMU will bypassed.
So the risk is to have an half secure system and may be unnoticed until later. I realize this is the current behavior, so not very ideal.
It feels to me if the user requested to use IOMMU then if we should panic if any of the available IOMMU are not initialized correctly. This will save a lot of debug afterwards.
@Stefano, any opinions?
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request
Quoted 169 lines · Julien Grall
Hi Oleksandr,
Quoted 138 lines · Oleksandr
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi, Julien
Quoted 47 lines · Oleksandr
Quoted 45 lines · Julien Grall
Quoted 43 lines · Oleksandr Tyshchenko
Quoted 33 lines
From: Oleksandr Tyshchenko <[email protected]>
This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.
In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen
-EAGAINto be used for indicating that device probing is deferred.This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.
Signed-off-by Oleksandr Tyshchenko <[email protected]>---
xen/common/device_tree.c | 1 +
xen/drivers/passthrough/arm/iommu.c| 35
++++++++++++++++++++++++++++++++++-
xen/include/asm-arm/device.h | 6 +++++-
xen/include/xen/device_tree.h | 1 +4 files changed, 41 insertions(+), 2 deletions(-)xen/common/device_tree.c +0
-0xen/common/device_tree.c static unsigned long __init · line 1778
Quoted 5 lines
xen/common/device_tree.c +1
-0@@ -1774,6 +1774,7 @@ static unsigned long __init unflatten_dt_node(const void *fdt, /* By default the device is not protected */ np->is_protected = false; INIT_LIST_HEAD(&np->domain_list); + INIT_LIST_HEAD(&np->deferred_probe);I am not entirely happy to add a new
list_headfield per node just for the benefits of boot code. Could we re-usedomain_list(with a comment in the code and appropriate ASSERT)?Quoted 8 lines
Quoted 3 lines
Agree that only boot code uses
deferred_probefield. I will consider re-usingdomain_list. Could you please clarify regarding ASSERT (where to put and what to check).What I meant is adding an ASSERT to check that
np->domain_listis at empty at least before trying to add in the list. This would help to debug any potential issue if we end up to usedomain_listearlier in the future. I can't see why it would as iommu is called earlier, but who knows :).Got it. Thank you for clarification.
Quoted 50 lines
Quoted 50 lines
Quoted 25 lines
Quoted 25 lines
xen/common/device_tree.c +1
-0+ static const struct iommu_ops *iommu_ops;const struct
iommu_ops*iommu_get_ops(void)xen/common/device_tree.c +1
-1@@ -42,7 +48,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops) int __init iommu_hardware_setup(void) { - struct dt_device_node *np; + struct dt_device_node *np, *tmp; int rc; unsigned int num_iommus = 0; @@ -51,6 +57,33 @@ int __init iommu_hardware_setup(void)rc =
device_init(np, DEVICE_IOMMU, NULL);
if ( !rc )
num_iommus++;
+ else if (rc ==-EAGAIN)
+ /*
+ * Driver requested deferred probing, so add this device to
+ * the deferred list for further processing. + */
+list_add(&np->deferred_probe, &deferred_probe_list); + }
+
+ /*Quoted 8 lines
Quoted 4 lines
+ * Process devices in the deferred list if at least one successfully
+ * probed device is present.
+ */I think this can turn into an infinite loop if all device in
deferred_probe_liststill return -EDEFER_PROBE andnum_iommusis a non-zero.Agree.
Quoted 4 lines
A better condition would be to check that at least one IOMMU is added at each loop. If not, then we should bail with an error because it likely means something is buggy.
Sounds reasonable. Will do.
Just to clarify:
>>> A better condition would be to check that at least one IOMMU is added at each loop.
Quoted 6 lines
Quoted 2 lines
Maybe, not only added (rc == 0), but driver didn't request deferred probe (rc !=
-EAGAIN).I think adding an IOMMU is enough. If you return an error other than
-EAGAINhere after deferring probing, then you are likely going to fail at the next loop. So better to stop early.It makes sense.
Quoted 9 lines
I realize this not what the current code is doing (I know I wrote it ;)). But I am not sure it is sane to continue if only part of the IOMMUs are initialized. Most likely you will see an error much later that may be not trivial to find out.
Imagine you want to passthrough you network card to a guest but the IOMMU initialization failed...
Oh, agree.
Quoted 8 lines
As I understand, the new strict logic would be the following:
If initialization for at least one IOMMU device failed (
device_initreturns an error other than-EAGAIN), we should stop and return an error to upper layer (even ifnum_iommus> 0). No matter whether it is during the first attempt or after deferring probe. We don't allow the "I/O virtualisation" to be enabled (iommu_enabled== true) with only part of the IOMMU devices being initialized. Is my understanding correct?Let me summarize the discussion we had on IRC :). Without your patch, Xen may initialize only half the IOMMUs. If the device is behind an IOMMU that wasn't initialized, then we have two possibility:
1) The device was already mark as protected (if using the old binding in
the SMMU). Xen will not be able to assign the device to Dom0 and therefore Xen will crash (not able to build dom0). For domU, it will depend whether the configuration contain the options 'dtdev'. If the option is specified, then guest will fail to build. On the contrary if the option isn't specified then the guest will boot and this could either lead to transaction failure (if the IOMMU was already reset) or bypassing the IOMMU. Note that the latter can today happen if your IOMMU was disabled. But we can't do much against it.
2) The device is not marked as protected. Xen will not be able to "assign"
the device to Dom0 and this could either lead to the device bypassing the IOMMU or a transaction failure. For domU, the problem is similar to 1).In the case of the SMMU driver, we only support old bindings. So devices are marked as protected during SMMU initialization. This is done before the SMMU is reset. Before reset the SMMU will bypassed.
Quoted 8 lines
So the risk is to have an half secure system and may be unnoticed until later. I realize this is the current behavior, so not very ideal.
It feels to me if the user requested to use IOMMU then if we should panic if any of the available IOMMU are not initialized correctly. This will save a lot of debug afterwards.
@Stefano, any opinions?
I agree that we should enable all IOMMUs or none. I don't think we want to deal with partially initialized IOMMUs systems.
Failure to enable all IOMMUs should lead to returning an error from the relevant function (arch_iommu_populate_page_table?) which should translate into Xen failing to boot and crashing. Which I think it is what you are suggesting, right?
(I wouldn't call panic() inside the IOMMU specific initializer, because there might be iommu= command line options for Xen that specify a different desired outcome. I would deal with this case the same way we would deal with an error during initialization of a single IOMMU.)
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request
Hi Stefano,
Quoted 6 lines · Stefano Stabellini
On Wed, 14 Aug 2019, Julien Grall wrote:
I agree that we should enable all IOMMUs or none. I don't think we want to deal with partially initialized IOMMUs systems.Failure to enable all IOMMUs should lead to returning an error from the relevant function (
arch_iommu_populate_page_table?) which should
The patch is:
|> start_xen() |> iommu_setup() |> iommu_hardware_setup()
Quoted 2 lines
translate into Xen failing to boot and crashing. Which I think it is what you are suggesting, right?
That's correct. At the moment the return value of iommu_setup() is ignored. What I would like to suggest is something along the following lines:
rc = iommu_setup();
if ( iommu_enable && rc != -ENODEV )
panic("Unable to setup IOMMUs");
Quoted 5 lines
(I wouldn't call
panic()inside the IOMMU specific initializer, because there might be iommu= command line options for Xen that specify a different desired outcome. I would deal with this case the same way we would deal with an error during initialization of a single IOMMU.)
I am not sure to understand this. If you have an half initialized IOMMU (note the "single" here!), then continuing is likely to make things much worst.
I don't advocate to put the panic() inside the IOMMU specific initializer (see above). But clearly, we should return an error no matter the content of 'iommu' command line if the user requested to enable the IOMMUs (if any). It wouldn't be right if the user can say "ignore IOMMU error" as most likely you will have unexpected error afterwards.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request
Hi,
Quoted 30 lines · Julien Grall
Quoted 6 lines · Stefano Stabellini
On Wed, 14 Aug 2019, Julien Grall wrote:
I agree that we should enable all IOMMUs or none. I don't think we want to deal with partially initialized IOMMUs systems.Failure to enable all IOMMUs should lead to returning an error from the relevant function (
arch_iommu_populate_page_table?) which shouldThe patch is:
|> start_xen() |> iommu_setup() |> iommu_hardware_setup()Quoted 2 lines
translate into Xen failing to boot and crashing. Which I think it is what you are suggesting, right?
That's correct. At the moment the return value of
iommu_setup()is ignored. What I would like to suggest is something along the following lines:rc =
iommu_setup();
if (iommu_enable&& rc !=-ENODEV)
panic("Unable to setup IOMMUs");Quoted 5 lines
(I wouldn't call
panic()inside the IOMMU specific initializer, because there might be iommu= command line options for Xen that specify a different desired outcome. I would deal with this case the same way we would deal with an error during initialization of a single IOMMU.)
Quoted 8 lines
I am not sure to understand this. If you have an half initialized IOMMU (note the "single" here!), then continuing is likely to make things much worst.
I don't advocate to put the
panic()inside the IOMMU specific initializer (see above). But clearly, we should return an error no matter the content of 'iommu' command line if the user requested to enable the IOMMUs (if any). It wouldn't be right if the user can say "ignore IOMMU error" as most likely you will have unexpected error afterwards.
I noticed there was already a panic() in iommu_setup() just in case the user force the use of IOMMU but they were not initialized. I was half-tempted to set iommu_force to true for Arm, but I think this is a different issue.
So here my take (not tested nor compiled).
xen/arch/arm/setup.c +4 -1
@@ -755,6 +755,7 @@ void __init start_xen(unsigned long boot_phys_offset, .max_grant_frames = gnttab_dom0_frames(), .max_maptrack_frames = opt_max_maptrack_frames, }; +int rc; dcache_line_bytes =read_dcache_line_bytes ();
@@ -890,7 +891,9 @@ void __init start_xen(unsigned long boot_phys_offset,setup_virt_paging (); -iommu_setup (); + rc =iommu_setup (); +if ( !iommu_enabled && rc != -ENODEV ) +panic ("Couldn't configure correctly all the IOMMUs." );do_initcalls ();
xen/drivers/passthrough/arm/iommu.c +8 -0
@@ -51,6 +51,14 @@ int __init iommu_hardware_setup(void) rc =device_init (np,DEVICE_IOMMU ,NULL );if ( !rc ) num_iommus++; +/* +* Ignore the following error codes: +* - EBADF: Indicate the current not is not an IOMMU +* - ENODEV: The IOMMU is not present or cannot be used by +* Xen. +*/ +else if ( rc != -EBADF && rc != -ENODEV ) +return rc; }return ( num_iommus >0 ) ?0 : -ENODEV ;
Quoted 3 lines
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request
Quoted 1 line · Julien Grall
Hi,
Hi Julien
Quoted 5 lines
I noticed there was already a
panic()iniommu_setup()just in case the user force the use of IOMMU but they were not initialized. I was half-tempted to setiommu_forceto true for Arm, but I think this is a different issue.So here my take (not tested nor compiled).
Thank you. I will check it and come back with results.
Quoted 49 lines
xen/arch/arm/setup.c +4
-1@@ -755,6 +755,7 @@ void __init start_xen(unsigned long boot_phys_offset, .max_grant_frames = gnttab_dom0_frames(), .max_maptrack_frames = opt_max_maptrack_frames, }; +int rc; dcache_line_bytes =read_dcache_line_bytes ();@@ -890,7 +891,9 @@ void __init start_xen(unsigned long boot_phys_offset,setup_virt_paging (); -iommu_setup (); + rc =iommu_setup (); +if ( !iommu_enabled && rc != -ENODEV ) +panic ("Couldn't configure correctly all the IOMMUs." );do_initcalls ();xen/drivers/passthrough/arm/iommu.c +8
-0@@ -51,6 +51,14 @@ int __init iommu_hardware_setup(void) rc =device_init (np,DEVICE_IOMMU ,NULL );if ( !rc ) num_iommus++; +/* +* Ignore the following error codes: +* - EBADF: Indicate the current not is not an IOMMU +* - ENODEV: The IOMMU is not present or cannot be used by +* Xen. +*/ +else if ( rc != -EBADF && rc != -ENODEV ) +return rc; }return ( num_iommus >0 ) ?0 : -ENODEV ;Quoted 6 lines
Cheers,
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 2/6] iommu/arm: Add ability to handle deferred probing request
Hi, Julien
Quoted 8 lines
Quoted 6 lines
I noticed there was already a
panic()iniommu_setup()just in case the user force the use of IOMMU but they were not initialized. I was half-tempted to setiommu_forceto true for Arm, but I think this is a different issue.So here my take (not tested nor compiled).
Thank you. I will check it and come back with results.
I have preliminary tested it with my IPMMU series including new modification of "deferred probing" patch (attached). Being honest my series is not based on the *current* staging, but not too outdated.
So, your patch works as expected in the following scenarios:
1. [No panic] Without IOMMU driver in Xen (#CONFIG_IPMMU_VMSA is not set).
2. [No panic] IOMMU driver is present, but reports it is cannot be used in Xen (P2M sharing not supported, etc) by returning -ENODEV.
3. [No panic] IOMMU is globally disabled in command line "iommu=0".
4. [No panic] IOMMU driver requests deferred probing until the last device in DT is initialized, after that all deferred devices get initialized.
5. [Panic] IOMMU driver returns an error (other than -EBADF and -ENODEV) at the very beginning/when a part of devices are already initialized.
6. [Panic] IOMMU driver always returns deferred probing/returns an error other than -EAGAIN after deferred probing.
Quoted 10 lines
Quoted 10 lines
xen/arch/arm/setup.c +0
-0@@ -755,6 +755,7 @@ void __init start_xen(unsigned long boot_phys_offset, .max_grant_frames = gnttab_dom0_frames(), .max_maptrack_frames = opt_max_maptrack_frames, };
Quoted 8 lines
Quoted 8 lines
xen/arch/arm/setup.c +1
-0+ int rc; dcache_line_bytes = read_dcache_line_bytes(); @@ -890,7 +891,9 @@ void __init start_xen(unsigned long boot_phys_offset,
setup_virt_paging();
-iommu_setup();
+ rc =iommu_setup();
+ if ( !iommu_enabled && rc !=-ENODEV)
+panic("Couldn't configure correctly all the IOMMUs.");
"\n" should be added.
Signature · 3 lines
Regards, Oleksandr Tyshchenko
From ff0af4d3d332a4b7b1b9b7edd4dbd1b3a263cb70 Mon Sep 17 00:00:00 2001
From: Oleksandr Tyshchenko <[email protected]>
Date: Thu, 15 Aug 2019 16:56:04 +0300
Subject: [PATCH] iommu/arm: Add ability to handle deferred probing request
This patch adds minimal required support to General IOMMU framework to be able to handle a case when IOMMU driver requesting deferred probing for a device.
In order not to pull Linux's error code (-EPROBE_DEFER) to Xen we have chosen -EAGAIN to be used for indicating that device probing is deferred.
This is needed for the upcoming IPMMU driver which may request deferred probing depending on what device will be probed the first (there is some dependency between these devices, Root device must be registered before Cache devices. If not the case, driver will deny further Cache device probes until Root device is registered). As we can't guarantee a fixed pre-defined order for the device nodes in DT, we need to be ready for the situation where devices being probed in "any" order.
While here, order the headers alphabetically.
---
Changes V2 -> V3:
- removed deferred_probe field from struct dt_device_node, re-used domain_list instead
- documented domain_list usage
- added ASSERT to check that np->domain_list is empty before re-using it
- put deferred_probe_list to init section
- used more strict logic regarding processing devices in the deferred list
- added more comments to code
- put headers in alphabetical order
--- xen/drivers/passthrough/arm/iommu.c | 55 ++++++++++++++++++++++++++++++++++--- xen/include/asm-arm/device.h | 6 +++- xen/include/xen/device_tree.h | 7 +++++ 3 files changed, 63 insertions(+), 5 deletions(-)
xen/drivers/passthrough/arm/iommu.c +51 -4
@@ -15,11 +15,20 @@* GNU General Public License for more details. */ -#include <xen/lib.h> -#include <xen/iommu.h> #include <xen/device_tree.h> +#include <xen/iommu.h> +#include <xen/lib.h> +#include <asm/device.h> +/* +* Deferred probe list is used to keep track of devices for which driver +* requested deferred probing (returned -EAGAIN). +* +* We re-use device's domain_list to link the device in the deferred list. +*/ +static __initdata LIST_HEAD (deferred_probe_list); +static const struct iommu_ops *iommu_ops;const struct iommu_ops *iommu_get_ops (void )
@@ -42,7 +51,7 @@ void __init iommu_set_ops(const struct iommu_ops *ops)int __init iommu_hardware_setup (void ) { -struct dt_device_node *np; +struct dt_device_node *np, *tmp;int rc;unsigned int num_iommus =0 ;
@@ -51,6 +60,17 @@ int __init iommu_hardware_setup(void) rc =device_init (np,DEVICE_IOMMU ,NULL );if ( !rc ) num_iommus++; +else if ( rc == -EAGAIN ) + { +/* We expect nobody uses domain_list at such early stage. */ +ASSERT (list_empty (&np->domain_list)); + +/* +* Driver requested deferred probing, so add this device to +* the deferred list for further processing. +*/ +list_add (&np->domain_list, &deferred_probe_list); + }/* * Ignore the following error codes: * - EBADF: Indicate the current is not an IOMMU
@@ -60,7 +80,34 @@ int __init iommu_hardware_setup(void)return rc; } -return ( num_iommus >0 ) ?0 : -ENODEV ; + rc = ( num_iommus >0 ) ?0 : -ENODEV ; + +/* +* Process devices in the deferred list if it is not empty and at least one +* successfully probed device is present. Check that at least one device +* is added at each loop, otherwise we may get an infinite loop. Also stop +* processing if we got an error other than -EAGAIN. +*/ +while ( !list_empty (&deferred_probe_list) && num_iommus ) + { + num_iommus =0 ; + + list_for_each_entry_safe ( np, tmp, &deferred_probe_list, domain_list ) + { + rc =device_init (np,DEVICE_IOMMU ,NULL ); +if ( !rc ) + { + num_iommus++; + +/* Remove successfully probed device from the deferred list. */ +list_del_init (&np->domain_list); + } +else if ( rc != -EAGAIN ) +return rc; + } + } + +return rc; } void __hwdom_init arch_iommu_check_autotranslated_hwdom(struct domain *d)
xen/include/asm-arm/device.h +5 -1
@@ -44,7 +44,11 @@ struct device_desc {enum device_class class;/* List of devices supported by this driver */ const struct dt_device_match *dt_match; -/* Device initialization */ +/* +* Device initialization. +* +* -EAGAIN is used to indicate that device probing is deferred. +*/ int (*init)(struct dt_device_node *dev,const void *data); };
xen/include/xen/device_tree.h +7 -0
@@ -92,6 +92,13 @@ struct dt_device_node {/* IOMMU specific fields */ bool is_protected; +/* +* The main purpose of this list node is to link the structure in the list +* of devices assigned to domain. +* +* Boot code (iommu_hardware_setup) re-uses this list to link the structure +* in the list of devices for which driver requested deferred probing. +*/ struct list_head domain_list;struct device dev;
Signature · 1 line
2.7.4
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH v2 3/6 xen/common: Introduce _xrealloc function
In reply to Oleksandr TyshchenkoFrom: Oleksandr Tyshchenko <[email protected]>
Next patch in this series will make use of it.
Original patch was initially posted by Sameer Goel:
https://lists.xen.org/archives/html/xen-devel/2017-06/msg00858.html
This could be considered as another attempt to add it:
https://www.mail-archive.com/[email protected]/msg21335.html
---
[As it was previously discussed with Julien in IRC]
The reason for this patch to be an RFC is that patch itself is not completely correct and I don't fully understand what/how should be done for this patch to be accepted. Or whether community even wants this to go in. So, to avoid bike shedding, the first target is to collect feedback from the maintainers.
In a nutshell, the upcoming "iommu_fwspec" support on ARM is going to use xrealloc when adding new device ID.
We really want to have "iommu_fwspec" support which will give us a generic abstract way to add new device to the IOMMU based on the generic IOMMU DT binding.
This is how Linux does:
https://github.com/torvalds/linux/blob/master/drivers/iommu/iommu.c#L2257
and we are doing the similar in next patch of this thread:
"iommu/arm: Add lightweight iommu_fwspec support"
--- xen/common/xmalloc_tlsf.c | 21 +++++++++++++++++++++ xen/include/xen/xmalloc.h | 1 + 2 files changed, 22 insertions(+)
xen/common/xmalloc_tlsf.c +21 -0
@@ -610,6 +610,27 @@ void *_xzalloc(unsigned long size, unsigned long align)return p ?memset (p,0 , size) : p; } +void *_xrealloc (void *p,unsigned long new_size,unsigned long align) +{ +void *new_p; + +if ( !new_size ) + { +xfree (p); +return NULL ; + } + + new_p =_xmalloc (new_size, align); + +if ( new_p && p ) + { +memcpy (new_p, p, new_size); +xfree (p); + } + +return new_p; +} +void xfree (void *p) {struct bhdr *b;
xen/include/xen/xmalloc.h +1 -0
@@ -51,6 +51,7 @@ extern void xfree(void *);/* Underlying functions */ extern void *_xmalloc (unsigned long size,unsigned long align);extern void *_xzalloc (unsigned long size,unsigned long align); +extern void *_xrealloc (void *p,unsigned long new_size,unsigned long align);static inline void *_xmalloc_array (unsigned long size,unsigned long align,unsigned long num)
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function
Quoted 16 lines · Oleksandr Tyshchenko
xen/common/xmalloc_tlsf.c +10
-0@@ -610,6 +610,27 @@ void *_xzalloc(unsigned long size, unsigned long align)return p ?memset (p,0 , size) : p; } +void *_xrealloc (void *p,unsigned long new_size,unsigned long align) +{ +void *new_p; + +if ( !new_size ) + { +xfree (p); +return NULL ; + } +
xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 632
Quoted 10 lines
xen/common/xmalloc_tlsf.c +10
-0+ new_p =_xmalloc (new_size, align); + +if ( new_p && p ) + { +memcpy (new_p, p, new_size); +xfree (p); + } + +return new_p; +}
While I can see why having a re-allocation function may be handy, explicit / direct use of _xmalloc() and _xzalloc() are discouraged, in favor of the more type-safe underscore-less variants. I can't see though how a type-safe "realloc" could look like, except for arrays. If resizing arrays is all you're after, I'd like to recommend to go that route rather then the suggested one here. If resizing arbitrary objects is the goal, then what you suggest may be the only route, but I'd still be not overly happy to see such added.
Furthermore you don't even use internals of the allocator: It is common practice to avoid re-allocation if the requested size fits within the already allocated block. That's not the least helpful because in such a case you can't possibly suffer any -ENOMEM condition.
And finally - please note _xmalloc()'s and _xfree()'s use / special casing of ZERO_BLOCK_PTR: You absolutely would need to mirror this here.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function
On 05.08.19 13:02, Jan Beulich wrote:
Hi, Jan
Quoted 3 lines
While I can see why having a re-allocation function may be handy, explicit / direct use of
_xmalloc()and_xzalloc()are discouraged, in favor of the more type-safe underscore-less variants.
took into account
Quoted 7 lines
I can't
see though how a type-safe "realloc" could look like, except for arrays. If resizing arrays is all you're after, I'd like to recommend to go that route rather then the suggested one here. If resizing arbitrary objects is the goal, then what you suggest may be the only route, but I'd still be not overly happy to see such added.
My main goal is to get "ported" from Linux "iommu_fwspec" support (xrealloc user) in [1].
I tried to retain code as much as possible while porting. So, this patch adds almost the same thing what the ported code expects.
But, I would be OK to consider modifying a code in a way to resize an array as well as any other variants if present.
Quoted 6 lines
Furthermore you don't even use internals of the allocator: It is common practice to avoid re-allocation if the requested size fits within the already allocated block. That's not the least helpful because in such a case you can't possibly suffer any
-ENOMEMcondition.
agree, took into account as well.
Quoted 4 lines
And finally - please note _xmalloc()'s and _xfree()'s use / special casing of
ZERO_BLOCK_PTR: You absolutely would need to mirror this here.
got it, will use for zero-size allocation
[1] https://lists.xenproject.org/archives/html/xen-devel/2019-08/msg00257.html
Thank you.
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function
Quoted 9 lines · Oleksandr
Quoted 7 lines · Jan Beulich
I can't
see though how a type-safe "realloc" could look like, except for arrays. If resizing arrays is all you're after, I'd like to recommend to go that route rather then the suggested one here. If resizing arbitrary objects is the goal, then what you suggest may be the only route, but I'd still be not overly happy to see such added.
Quoted 5 lines
My main goal is to get "ported" from Linux "
iommu_fwspec" support (xrealloc user) in [1].I tried to retain code as much as possible while porting. So, this patch adds almost the same thing what the ported code expects.
But, I would be OK to consider modifying a code in a way to resize an array as well as any other variants if present.
I've looked at the use in patch 4, and it really isn't an array allocation. Even a basic allocation would use _xmalloc() in this case (you'll find examples in the tree if you want). Nevertheless I'd appreciate if the type-unsafe _xrealloc() didn't remain the only re-allocation construct, as to avoiding people using it just because there's no better alternative.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function
Hi,
Quoted 4 lines
Nevertheless
I'd appreciate if the type-unsafe_xrealloc()didn't remain the only re-allocation construct, as to avoiding people using it just because there's no better alternative.
I got your point.
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function
In reply to Jan BeulichHi Jan,
Quoted 29 lines · Jan Beulich
Quoted 16 lines · Oleksandr Tyshchenko
xen/common/xmalloc_tlsf.c +10
-0@@ -610,6 +610,27 @@ void *_xzalloc(unsigned long size, unsigned long align)return p ?memset (p,0 , size) : p; } +void *_xrealloc (void *p,unsigned long new_size,unsigned long align) +{ +void *new_p; + +if ( !new_size ) + { +xfree (p); +return NULL ; + } +xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 632
Quoted 10 lines
xen/common/xmalloc_tlsf.c +10
-0+ new_p =_xmalloc (new_size, align); + +if ( new_p && p ) + { +memcpy (new_p, p, new_size); +xfree (p); + } + +return new_p; +}While I can see why having a re-allocation function may be handy,
Quoted 8 lines
explicit / direct use of
_xmalloc()and_xzalloc()are discouraged, in favor of the more type-safe underscore-less variants. I can't see though how a type-safe "realloc" could look like, except for arrays. If resizing arrays is all you're after, I'd like to recommend to go that route rather then the suggested one here. If resizing arbitrary objects is the goal, then what you suggest may be the only route, but I'd still be not overly happy to see such added.
I can see 3 uses for realloc:
a. re-allocate generic data buffer
b. re-allocate array
c. re-allocate struct with flexible buffer.
option c. is about structures like this:
struct arrlen
{
size_t len;
int data[1];
};This is Oleksandr's case.
So for option a. we can use _xreallocate(ptr, size, align) For option b. we can use xrealloc_array(_ptr, _type, _num) And for option c. I propose to implement the following macro:
#define realloc_flex_struct(_ptr, _type, _field, _len) \
((_type *)_xrealloc(_ptr, offsetof(_type, _field[_len]) , __alignof__(_type)))
It can be used in the following way:
newptr = realloc_flex_struct(ptr, struct arrlen, newsize);
As you can see, this approach is type-safe and covers Oleksanrd's case.
Signature · 5 lines
Volodymyr Babchuk at EPAM _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function
Quoted 63 lines · Volodymyr Babchuk
Hi Jan,
Quoted 29 lines · Jan Beulich
Quoted 16 lines · Oleksandr Tyshchenko
xen/common/xmalloc_tlsf.c +10
-0@@ -610,6 +610,27 @@ void *_xzalloc(unsigned long size, unsigned long align)return p ?memset (p,0 , size) : p; } +void *_xrealloc (void *p,unsigned long new_size,unsigned long align) +{ +void *new_p; + +if ( !new_size ) + { +xfree (p); +return NULL ; + } +xen/common/xmalloc_tlsf.c void *_xzalloc(unsigned long size, unsigned long align) · line 632
Quoted 10 lines
xen/common/xmalloc_tlsf.c +10
-0+ new_p =_xmalloc (new_size, align); + +if ( new_p && p ) + { +memcpy (new_p, p, new_size); +xfree (p); + } + +return new_p; +}While I can see why having a re-allocation function may be handy,
Quoted 8 lines
explicit / direct use of
_xmalloc()and_xzalloc()are discouraged, in favor of the more type-safe underscore-less variants. I can't see though how a type-safe "realloc" could look like, except for arrays. If resizing arrays is all you're after, I'd like to recommend to go that route rather then the suggested one here. If resizing arbitrary objects is the goal, then what you suggest may be the only route, but I'd still be not overly happy to see such added.I can see 3 uses for realloc:
a. re-allocate generic data buffer
b. re-allocate array
c. re-allocate struct with flexible buffer.option c. is about structures like this:
struct arrlen
{ size_t len; int data[1]; };This is Oleksandr's case.
So for option a. we can use
_xreallocate(ptr, size, align)For option b. we can usexrealloc_array(_ptr, _type, _num)And for option c. I propose to implement the following macro:
Quoted 8 lines
#define
realloc_flex_struct(_ptr, _type, _field, _len)\
((_type*)_xrealloc(_ptr,offsetof(_type, _field[_len]),__alignof__(_type)))It can be used in the following way:
newptr =
realloc_flex_struct(ptr, struct arrlen, newsize);As you can see, this approach is type-safe and covers Oleksanrd's case.
This looks fine to me, but then wants to be accompanied by a similar xmalloc_flex_struct(), which could be used right away to replace a number of open-coded instances of the above.
There's one more thing for the re-alloc case though (besides cosmetic aspects): The incoming pointer should also be verified to be of correct type.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function
Hi, Jan, Volodymyr.
Quoted 21 lines
Quoted 21 lines
c. re-allocate struct with flexible buffer.
option c. is about structures like this:
struct arrlen
{
size_tlen;
int data[1];
};This is Oleksandr's case.
So for option a. we can use
_xreallocate(ptr, size, align)For option b. we can usexrealloc_array(_ptr, _type, _num)And for option c. I propose to implement the following macro:#define
realloc_flex_struct(_ptr, _type, _field, _len)\
((_type*)_xrealloc(_ptr,offsetof(_type, _field[_len]),__alignof__(_type)))It can be used in the following way:
Quoted 7 lines
Quoted 3 lines
newptr =
realloc_flex_struct(ptr, struct arrlen, newsize);As you can see, this approach is type-safe and covers Oleksanrd's case.
This looks fine to me, but then wants to be accompanied by a similar
xmalloc_flex_struct(), which could be used right away to replace a number of open-coded instances of the above.
Thank you Volodymyr for the idea. Looks like we can get a type-safe approach which looks suitable for my particular case.
So, I need to focus on the proper implementation of non type-safe (_xrealloc) variant in the first place taking into the account Jan's comments. Then I will be back to the suggested type-safe marco (realloc_flex_struct).
Quoted 4 lines
There's one more thing for the re-alloc case though (besides cosmetic aspects): The incoming pointer should also be verified to be of correct type.
Jan, how this could be technically implemented, or are these any existing examples in Xen?
Quoted 2 lines
Jan
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function
Quoted 5 lines · Oleksandr
Quoted 3 lines
There's one more thing for the re-alloc case though (besides cosmetic aspects): The incoming pointer should also be verified to be of correct type.
Jan, how this could be technically implemented, or are these any existing examples in Xen?
See x86's copy_to_guest_offset(), for example. To get the compiler to emit a warning (at least), a (typically otherwise dead) comparison of pointers is commonly used.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function
In reply to Oleksandr(I'm sorry if you receive duplicates of this, but I've got a reply back from our mail system that several of the recipients did not have their host names resolved correctly on the first attempt.)
Quoted 5 lines · Oleksandr
Quoted 3 lines
There's one more thing for the re-alloc case though (besides cosmetic aspects): The incoming pointer should also be verified to be of correct type.
Jan, how this could be technically implemented, or are these any existing examples in Xen?
See x86's copy_to_guest_offset(), for example. To get the compiler to emit a warning (at least), a (typically otherwise dead) comparison of pointers is commonly used.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 3/6] [RFC] xen/common: Introduce _xrealloc function
On 08.08.19 10:05, Jan Beulich wrote:
Hi Jan
Quoted 3 lines
(I'm sorry if you receive duplicates of this, but I've got a reply back from our mail system that several of the recipients did not have their host names resolved correctly on the first attempt.)
Absolutely no problem.
Quoted 4 lines
Quoted 1 line
Jan, how this could be technically implemented, or are these any existing examples in Xen?
See x86's
copy_to_guest_offset(), for example. To get the compiler to emit a warning (at least), a (typically otherwise dead) comparison of pointers is commonly used.
Thank you for the pointer. It is clear now.
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 4/6 iommu/arm: Add lightweight iommu_fwspec support
In reply to Oleksandr TyshchenkoFrom: Oleksandr Tyshchenko <[email protected]>
We need to have some abstract way to add new device to the IOMMU based on the generic IOMMU DT binding [1] which can be used for both DT (right now) and ACPI (in future).
For that reason we can borrow the idea used in Linux these days called "iommu_fwspec". Having this in, it will be possible to configure IOMMU master interfaces of the device (device IDs) from a single common place and avoid keeping almost identifical look-up implementations in each IOMMU driver.
There is no need to port the whole implementation of "iommu_fwspec" to Xen, we could, probably, end up with a much simpler solution, some "stripped down" version which fits our requirments.
So, this patch adds the following:
1. A common structure "iommu_fwspec" to hold the the per-device firmware data
2. New member "iommu_fwspec" of struct device
3. Functions/helpers to deal with "dev->iommu_fwspec"
It should be noted that in comparing with original "iommu_fwspec" Xen's variant doesn't contain some fields, which are not really needed at the moment (ops, flag) and "iommu_fwnode" field was replaced by "iommu_dev" to avoid porting a lot of code (to support "fwnode_handle") with little benefit.
Next patch in this series will make use of that support.
[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt
--- xen/drivers/passthrough/arm/Makefile | 2 +- xen/drivers/passthrough/arm/iommu_fwspec.c | 91 ++++++++++++++++++++++++++++++ xen/include/asm-arm/device.h | 1 + xen/include/asm-arm/iommu.h | 2 + xen/include/asm-arm/iommu_fwspec.h | 65 +++++++++++++++++++++ 5 files changed, 160 insertions(+), 1 deletion(-) create mode 100644 xen/drivers/passthrough/arm/iommu_fwspec.c create mode 100644 xen/include/asm-arm/iommu_fwspec.h
xen/drivers/passthrough/arm/Makefile +1 -1
@@ -1,2 +1,2 @@ -obj-y += iommu.o iommu_helpers.o +obj-y += iommu.o iommu_helpers.o iommu_fwspec.o obj-$(CONFIG_ARM_SMMU) += smmu.o
xen/drivers/passthrough/arm/iommu_fwspec.c new file +91 -0
91 lines not on this page · raw message
xen/include/asm-arm/device.h +1 -0
@@ -18,6 +18,7 @@ struct devicestruct dt_device_node *of_node;/* Used by drivers imported from Linux */ #endif struct dev_archdata archdata; +struct iommu_fwspec *iommu_fwspec;/* per-device IOMMU instance data */ };typedef struct device device_t ;
xen/include/asm-arm/iommu.h +2 -0
@@ -14,6 +14,8 @@#ifndef __ARCH_ARM_IOMMU_H__#define __ARCH_ARM_IOMMU_H__ +#include <asm/iommu_fwspec.h> +struct arch_iommu {/* Private information for the IOMMU drivers */
xen/include/asm-arm/iommu_fwspec.h new file +65 -0
65 lines not on this page · raw message
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support
Hi Oleksandr,
Quoted 10 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
We need to have some abstract way to add new device to the IOMMU based on the generic IOMMU DT binding [1] which can be used for both DT (right now) and ACPI (in future).
For that reason we can borrow the idea used in Linux these days called "
iommu_fwspec". Having this in, it will be possible to configure IOMMU master interfaces of the device (device IDs) from a single common place and avoid keeping almost identifical look-up
s/identifical/identical/
Quoted 5 lines
implementations in each IOMMU driver.
There is no need to port the whole implementation of "
iommu_fwspec" to Xen, we could, probably, end up with a much simpler solution, some "stripped down" version which fits our requirments.
s/requirments/requirements/
Quoted 57 lines
So, this patch adds the following:
1. A common structure "iommu_fwspec" to hold the the per-device firmware data
2. New member "iommu_fwspec" of struct device
3. Functions/helpers to deal with "dev->iommu_fwspec"It should be noted that in comparing with original "
iommu_fwspec" Xen's variant doesn't contain some fields, which are not really needed at the moment (ops, flag) and "iommu_fwnode" field was replaced by "iommu_dev" to avoid porting a lot of code (to support "fwnode_handle") with little benefit.Next patch in this series will make use of that support.
[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt
Signed-off-by Oleksandr Tyshchenko <[email protected]>--- xen/drivers/passthrough/arm/Makefile | 2 +- xen/drivers/passthrough/arm/iommu_fwspec.c | 91 ++++++++++++++++++++++++++++++ xen/include/asm-arm/device.h | 1 + xen/include/asm-arm/iommu.h | 2 + xen/include/asm-arm/iommu_fwspec.h | 65 +++++++++++++++++++++ 5 files changed, 160 insertions(+), 1 deletion(-)create mode 100644
xen/drivers/passthrough/arm/iommu_fwspec.ccreate mode 100644xen/include/asm-arm/iommu_fwspec.hxen/drivers/passthrough/arm/Makefile +1
-1@@ -1,2 +1,2 @@ -obj-y += iommu.o iommu_helpers.o +obj-y += iommu.o iommu_helpers.o iommu_fwspec.o obj-$(CONFIG_ARM_SMMU) += smmu.oxen/drivers/passthrough/arm/iommu_fwspec.c new file +15
-0@@ -0,0 +1,91 @@ +/* +* xen/drivers/passthrough/arm/iommu_fwspec.c +* +* Contains functions to maintain per-device firmware data +* +* Based on Linux's iommu_fwspec support you can find at: +* drivers/iommu/iommu.c +* +* Copyright (C) 2019 EPAM Systems Inc. +* +* This program is free software; you can redistribute it and/or +* modify it under the terms and conditions of the GNU General Public +* License, version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful,
xen/drivers/passthrough/arm/iommu_fwspec.c line 25
Quoted 10 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +10
-0+* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public +* License along with this program; If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/lib.h> +#include <xen/iommu.h>
Please order the headers alphabetically.
NIT: Can you a newline between xen and asm headers?
Quoted 16 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +15
-0+#include <asm/device.h> +#include <asm/iommu_fwspec.h> + +int iommu_fwspec_init (struct device *dev,struct device *iommu_dev) +{ +struct iommu_fwspec *fwspec =dev_iommu_fwspec_get (dev); + +if ( fwspec ) +return 0 ; + + fwspec = xzalloc(struct iommu_fwspec); +if ( !fwspec ) +return -ENOMEM ; + + fwspec->iommu_dev = iommu_dev;
xen/drivers/passthrough/arm/iommu_fwspec.c line 50
Quoted 10 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +10
-0+dev_iommu_fwspec_set (dev, fwspec); + +return 0 ; +} + +void iommu_fwspec_free (struct device *dev) +{ +struct iommu_fwspec *fwspec =dev_iommu_fwspec_get (dev); + +if ( fwspec )
xfree is able to deal with NULL pointer, so the check is not necessary.
xen/drivers/passthrough/arm/iommu_fwspec.c line 57
Quoted 7 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +7
-0+ { +xfree (fwspec); +dev_iommu_fwspec_set (dev,NULL ); + } +} + +int iommu_fwspec_add_ids (struct device *dev,uint32_t *ids,int num_ids)
While I realize the prototype is coming from Linux, num_ids cannot be negative (the code below would not work properly). So the parameter should be unsigned.
xen/drivers/passthrough/arm/iommu_fwspec.c line 61
Quoted 4 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +4
-0+{ +struct iommu_fwspec *fwspec =dev_iommu_fwspec_get (dev); +size_t size; +int i;
Any variable that can't be negative should be unsigned.
Quoted 47 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +30
-0+ +if ( !fwspec ) +return -EINVAL ; + + size = offsetof(struct iommu_fwspec, ids[fwspec->num_ids + num_ids]); +if ( size >sizeof (*fwspec) ) + { + fwspec =_xrealloc (fwspec, size,sizeof (void *)); +if ( !fwspec ) +return -ENOMEM ; + +dev_iommu_fwspec_set (dev, fwspec); + } + +for ( i =0 ; i < num_ids; i++ ) + fwspec->ids[fwspec->num_ids + i] = ids[i]; + + fwspec->num_ids += num_ids; + +return 0 ; +} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/ xen/include/asm-arm/device.h +1
-0@@ -18,6 +18,7 @@ struct devicestruct dt_device_node *of_node;/* Used by drivers imported from Linux */ #endif struct dev_archdata archdata; +struct iommu_fwspec *iommu_fwspec;/* per-device IOMMU instance data */ };typedef struct device device_t ;xen/include/asm-arm/iommu.h +0
-0
xen/include/asm-arm/iommu.h line 17
Quoted 4 lines
xen/include/asm-arm/iommu.h +1
-0@@ -14,6 +14,8 @@#ifndef __ARCH_ARM_IOMMU_H__#define __ARCH_ARM_IOMMU_H__ +#include <asm/iommu_fwspec.h>
iommu.h does not seem to use anything defined in iommu_fwspec.h. So why do you include it here?
Quoted 30 lines
xen/include/asm-arm/iommu.h +1
-0+struct arch_iommu {/* Private information for the IOMMU drivers */ xen/include/asm-arm/iommu_fwspec.h new file +20
-0@@ -0,0 +1,65 @@ +/* +* xen/include/asm-arm/iommu_fwspec.h +* +* Contains a common structure to hold the per-device firmware data and +* declaration of functions used to maintain that data +* +* Based on Linux's iommu_fwspec support you can find at: +* include/linux/iommu.h +* +* Copyright (C) 2019 EPAM Systems Inc. +* +* This program is free software; you can redistribute it and/or +* modify it under the terms and conditions of the GNU General Public +* License, version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +*
xen/include/asm-arm/iommu_fwspec.h line 30
Quoted 10 lines
xen/include/asm-arm/iommu_fwspec.h new file +10
-0+* You should have received a copy of the GNU General Public +* License along with this program; If not, see <http://www.gnu.org/licenses/>. +*/ + +#ifndef __ARCH_ARM_IOMMU_FWSPEC_H__ +#define __ARCH_ARM_IOMMU_FWSPEC_H__ + +/* per-device IOMMU instance data */ +struct iommu_fwspec { +/* device which represents this IOMMU H/W */
Did you intend to say "this device's IOMMU"?
Quoted 26 lines
xen/include/asm-arm/iommu_fwspec.h new file +25
-0+struct device *iommu_dev; +/* IOMMU driver private data for this device */ +void *iommu_priv; +/* number of associated device IDs */ +unsigned int num_ids; +/* IDs which this device may present to the IOMMU */ + uint32_t ids[1]; +}; + +int iommu_fwspec_init (struct device *dev,struct device *iommu_dev); +void iommu_fwspec_free (struct device *dev); +int iommu_fwspec_add_ids (struct device *dev,uint32_t *ids,int num_ids); + +static inline struct iommu_fwspec *dev_iommu_fwspec_get (struct device *dev) +{ +return dev->iommu_fwspec; +} + +static inline void dev_iommu_fwspec_set (struct device *dev, +struct iommu_fwspec *fwspec) +{ + dev->iommu_fwspec = fwspec; +} + +#endif /* __ARCH_ARM_IOMMU_FWSPEC_H__ */
xen/include/asm-arm/iommu_fwspec.h line 65
Quoted 10 lines
xen/include/asm-arm/iommu_fwspec.h new file +10
-0+ +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi Julien.
Quoted 8 lines · Oleksandr Tyshchenko
Quoted 6 lines
From: Oleksandr Tyshchenko <[email protected]>
We need to have some abstract way to add new device to the IOMMU based on the generic IOMMU DT binding [1] which can be used for both DT (right now) and ACPI (in future).
Quoted 6 lines
Quoted 4 lines
For that reason we can borrow the idea used in Linux these days called "
iommu_fwspec". Having this in, it will be possible to configure IOMMU master interfaces of the device (device IDs) from a single common place and avoid keeping almost identifical look-ups/identifical/identical/
ok
Quoted 8 lines
Quoted 6 lines
implementations in each IOMMU driver.
There is no need to port the whole implementation of "
iommu_fwspec" to Xen, we could, probably, end up with a much simpler solution, some "stripped down" version which fits our requirments.s/requirments/requirements/
ok
Quoted 9 lines
xen/drivers/passthrough/arm/iommu_fwspec.c line 25
Quoted 5 lines
+ */
xen/drivers/passthrough/arm/iommu_fwspec.c new file +3
-0+ +#include <xen/lib.h> +#include <xen/iommu.h> Please order the headers alphabetically.
NIT: Can you a newline between xen and asm headers?
Will do
Quoted 21 lines
Quoted 21 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +19
-0+#include <asm/device.h> +#include <asm/iommu_fwspec.h> + +int iommu_fwspec_init (struct device *dev,struct device *iommu_dev) +{ + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); + + if ( fwspec ) + return 0; + + fwspec = xzalloc(struct iommu_fwspec); + if ( !fwspec ) + return -ENOMEM; + + fwspec->iommu_dev = iommu_dev; + dev_iommu_fwspec_set(dev, fwspec); + + return 0; +}
Quoted 8 lines
xen/drivers/passthrough/arm/iommu_fwspec.c line 50
Quoted 6 lines
xen/drivers/passthrough/arm/iommu_fwspec.c new file +6
-0+ +void iommu_fwspec_free (struct device *dev) +{ + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); + + if ( fwspec )xfree is able to deal with NULL pointer, so the check is not necessary.
Yes, the reason I left this check is to not perform an extra operation (dev_iommu_fwspec_set). Shall I drop this check anyway?
Quoted 11 lines
Quoted 9 lines
+ {
+ xfree(fwspec);
+dev_iommu_fwspec_set(dev, NULL);
+ }xen/drivers/passthrough/arm/iommu_fwspec.c new file +3
-0+} + +int iommu_fwspec_add_ids(struct device *dev, uint32_t *ids, int
num_ids)While I realize the prototype is coming from Linux,
num_idscannot be negative (the code below would not work properly). So the parameter should be unsigned.
Agree, will use unsigned.
Quoted 7 lines
Quoted 5 lines
+{
+ structiommu_fwspec*fwspec =dev_iommu_fwspec_get(dev); +size_tsize;
+ int i;Any variable that can't be negative should be unsigned.
Yes, will follow.
Quoted 10 lines
Quoted 8 lines
xen/include/asm-arm/iommu.h +0
-0@@ -14,6 +14,8 @@ #ifndef __ARCH_ARM_IOMMU_H__ #define __ARCH_ARM_IOMMU_H__+#include <
asm/iommu_fwspec.h>
iommu.hdoes not seem to use anything defined iniommu_fwspec.h. So why do you include it here?
I was thinking that every source which includes iommu.h will get iommu_fwspec.h included indirectly. No need to include iommu_fwspec.h in many sources.
This was a reason. Shall I included it directly where needed?
Quoted 36 lines
Quoted 36 lines
+
structarch_iommu
{
/* Private information for the IOMMU drivers */xen/include/asm-arm/iommu_fwspec.h new file +22
-0@@ -0,0 +1,65 @@ +/* +* xen/include/asm-arm/iommu_fwspec.h +* +* Contains a common structure to hold the per-device firmware data and +* declaration of functions used to maintain that data +* +* Based on Linux's iommu_fwspec support you can find at: +* include/linux/iommu.h +* +* Copyright (C) 2019 EPAM Systems Inc. +* +* This program is free software; you can redistribute it and/or +* modify it under the terms and conditions of the GNU General Public +* License, version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public +* License along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
Quoted 8 lines
xen/include/asm-arm/iommu_fwspec.h line 30
Quoted 6 lines
xen/include/asm-arm/iommu_fwspec.h new file +6
-0+#ifndef __ARCH_ARM_IOMMU_FWSPEC_H__ +#define __ARCH_ARM_IOMMU_FWSPEC_H__ + +/* per-device IOMMU instance data */ +struct iommu_fwspec { + /* device which represents this IOMMU H/W */Did you intend to say "this device's IOMMU"?
Exactly)
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support
Hi,
Quoted 6 lines · Oleksandr
Quoted 2 lines · Julien Grall
xfree is able to deal with NULL pointer, so the check is not necessary.
Yes, the reason I left this check is to not perform an extra operation (
dev_iommu_fwspec_set). Shall I drop this check anyway?
I can't see any issue to do the extra operation. This is not hotpath and it is harmless.
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
xen/include/asm-arm/iommu.h +0
-0@@ -14,6 +14,8 @@ #ifndef __ARCH_ARM_IOMMU_H__ #define __ARCH_ARM_IOMMU_H__+#include <
asm/iommu_fwspec.h>
iommu.hdoes not seem to use anything defined iniommu_fwspec.h. So why do you include it here?
Quoted 3 lines
I was thinking that every source which includes
iommu.hwill getiommu_fwspec.hincluded indirectly. No need to includeiommu_fwspec.hin many sources.This was a reason. Shall I included it directly where needed?
There are a few cases where iommu.h is required but not iommu_fwspec.h. In general, I would prefer if headers are only included when strictly necessary.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support
Quoted 1 line · Julien Grall
Hi,
Hi Julien
Quoted 10 lines · Oleksandr
Quoted 6 lines · Julien Grall
Quoted 2 lines
xfree is able to deal with NULL pointer, so the check is not necessary.
Yes, the reason I left this check is to not perform an extra operation (
dev_iommu_fwspec_set). Shall I drop this check anyway?I can't see any issue to do the extra operation. This is not hotpath and it is harmless.
ok, will drop.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 10 lines
xen/include/asm-arm/iommu.h +0
-0@@ -14,6 +14,8 @@ #ifndef __ARCH_ARM_IOMMU_H__ #define __ARCH_ARM_IOMMU_H__+#include <
asm/iommu_fwspec.h>
iommu.hdoes not seem to use anything defined iniommu_fwspec.h. So why do you include it here?
Quoted 5 lines
Quoted 3 lines
I was thinking that every source which includes
iommu.hwill getiommu_fwspec.hincluded indirectly. No need to includeiommu_fwspec.hin many sources.This was a reason. Shall I included it directly where needed?
There are a few cases where
iommu.his required but notiommu_fwspec.h. In general, I would prefer if headers are only included when strictly necessary.
got it, will drop from here and include where necessary.
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support
In reply to Oleksandr TyshchenkoHi Oleksandr,
One more comment :).
xen/drivers/passthrough/arm/iommu_fwspec.c line 33
Quoted 5 lines · Oleksandr Tyshchenko
xen/drivers/passthrough/arm/iommu_fwspec.c new file +5
-0+int iommu_fwspec_init (struct device *dev,struct device *iommu_dev) +{ +struct iommu_fwspec *fwspec =dev_iommu_fwspec_get (dev); + +if ( fwspec )
I would actually check the iommu_dev passed in parameter is the same as stored. We expect all device to be protected by only one IOMMU. So better to be safe than allow overriding ;).
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 4/6] iommu/arm: Add lightweight iommu_fwspec support
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi Julien.
Quoted 11 lines
One more comment :).
xen/drivers/passthrough/arm/iommu_fwspec.c line 33
Quoted 5 lines · Oleksandr Tyshchenko
xen/drivers/passthrough/arm/iommu_fwspec.c new file +5
-0+int iommu_fwspec_init (struct device *dev,struct device *iommu_dev) +{ + struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev); + + if ( fwspec )I would actually check the
iommu_devpassed in parameter is the same as stored. We expect all device to be protected by only one IOMMU. So better to be safe than allow overriding ;).
Actually, it makes sense, will add appropriate check.
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 5/6 iommu/arm: Introduce iommu_add_dt_device API
In reply to Oleksandr TyshchenkoFrom: Oleksandr Tyshchenko <[email protected]>
This patch adds new iommu_add_dt_device API for adding DT device to the IOMMU using generic IOMMU DT binding [1] and previously added "iommu_fwspec" support.
New function parses the DT binding, prepares "dev->iommu_fwspec" with correct information and calls the IOMMU driver using "add_device" callback to register new DT device.
The IOMMU driver's responsibility is to check whether "dev->iommu_fwspec" is initialized and mark that device as protected.
The additional benefit here is to avoid to go through the whole DT multiple times in IOMMU driver trying to locate master devices which belong to each IOMMU device being probed.
The upcoming IPMMU driver will have "add_device" callback implemented.
I hope, this patch won't break SMMU driver's functionality, which doesn't have this callback implemented.
[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt
--- xen/arch/arm/domain_build.c | 12 ++++++++++ xen/drivers/passthrough/arm/iommu.c | 45 +++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/iommu.h | 3 +++ 3 files changed, 60 insertions(+)
xen/arch/arm/domain_build.c +12 -0
@@ -1241,6 +1241,18 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev,u64 addr, size;bool need_mapping = !dt_device_for_passthrough (dev); +if (dt_parse_phandle (dev,"iommus" ,0 ) ) + { +dt_dprintk ("%s add to iommu\n" ,dt_node_full_name (dev)); + res =iommu_add_dt_device (dev); +if ( res ) + { +printk (XENLOG_ERR "Failed to add %s to the IOMMU\n" , +dt_node_full_name (dev)); +return res; + } + } + nirq =dt_number_of_irq (dev); naddr =dt_number_of_address (dev);
xen/drivers/passthrough/arm/iommu.c +45 -0
48 lines not on this page · raw message
xen/include/asm-arm/iommu.h +3 -0
@@ -28,6 +28,9 @@ struct arch_iommuconst struct iommu_ops *iommu_get_ops (void );void iommu_set_ops (const struct iommu_ops *ops); +/* helper to add DT device to the IOMMU */ +int iommu_add_dt_device (struct dt_device_node *np); +/* mapping helpers */ int __must_check arm_iommu_map_page(struct domain *d, dfn_t dfn, mfn_t mfn, unsigned int flags,
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 5/6] iommu/arm: Introduce iommu_add_dt_device API
Hi Oleksandr,
Quoted 12 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
This patch adds new
iommu_add_dt_deviceAPI for adding DT device to the IOMMU using generic IOMMU DT binding [1] and previously added "iommu_fwspec" support.New function parses the DT binding, prepares "
dev->iommu_fwspec" with correct information and calls the IOMMU driver using "add_device" callback to register new DT device.
The IOMMU driver's responsibility is to check whether "dev->iommu_fwspec" is initialized and mark that device as protected.
Quoted 8 lines
The additional benefit here is to avoid to go through the whole DT multiple times in IOMMU driver trying to locate master devices which belong to each IOMMU device being probed.
The upcoming IPMMU driver will have "
add_device" callback implemented.I hope, this patch won't break SMMU driver's functionality, which doesn't have this callback implemented.
The last two sentence does not really belong to the commit message. So I think they should go after ---.
Anyway, the only concern for the SMMU is to not break the old bindings. New bindings are not supported, so it does not matter whether they are broken or not. Once this series is merged, we can have a look how new bindings for the SMMU can be supported.
Quoted 15 lines
[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt
Signed-off-by Oleksandr Tyshchenko <[email protected]>--- xen/arch/arm/domain_build.c | 12 ++++++++++ xen/drivers/passthrough/arm/iommu.c | 45 +++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/iommu.h | 3 +++ 3 files changed, 60 insertions(+)xen/arch/arm/domain_build.c +0
-0
xen/arch/arm/domain_build.c static int __init handle_device(struct domain *d, struct dt_device_node *dev, · line 1244
Quoted 4 lines
xen/arch/arm/domain_build.c +1
-0@@ -1241,6 +1241,18 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev,u64 addr, size;bool need_mapping = !dt_device_for_passthrough (dev); +if (dt_parse_phandle (dev,"iommus" ,0 ) )
I don't particularly like this check. dt_parse_phandle is non-trivial to execute.
TBH, what we should do is trying to call iommu_add_dt_device if IOMMU is enabled. We can then return a recognizable value to tell the device is not protected.
Quoted 22 lines
xen/arch/arm/domain_build.c +11
-0+ { +dt_dprintk ("%s add to iommu\n" ,dt_node_full_name (dev)); + res =iommu_add_dt_device (dev); +if ( res ) + { +printk (XENLOG_ERR "Failed to add %s to the IOMMU\n" , +dt_node_full_name (dev)); +return res; + } + } + nirq =dt_number_of_irq (dev); naddr =dt_number_of_address (dev);xen/drivers/passthrough/arm/iommu.c +0
-0@@ -113,3 +113,48 @@ int arch_iommu_populate_page_table(struct domain *d) void __hwdom_init arch_iommu_hwdom_init(struct domain *d) { }
xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 125
Quoted 10 lines
xen/drivers/passthrough/arm/iommu.c +10
-0+ +int __init iommu_add_dt_device (struct dt_device_node *np) +{ +const struct iommu_ops *ops =iommu_get_ops (); +struct dt_phandle_args iommu_spec; +struct device *dev =dt_to_dev (np); +int rc =1 , index =0 ; + +if ( !iommu_enabled || !ops || !ops->add_device ) +return 0 ;
While I agree that !iommu_enabled should return 0, for the two others I am not entirely sure this is the right thing to do.
!ops is definitely an error because if you have the IOMMU enabled then you should have ops installed.
!ops->add_device means that you will not be able to add any device using the new bindings. IOW, the device will be unusable later one as most likely the IOMMU was configured to deny any transaction. Therefore, this should return an error.
xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 130
Quoted 5 lines
xen/drivers/passthrough/arm/iommu.c +5
-0+ +if (dev_iommu_fwspec_get (dev) ) +return -EEXIST ; + +/* According to the Documentation/devicetree/bindings/iommu/iommu.txt */
This file does not exist in Xen, so Linux should at least be mentioned in the comment.
xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 141
Quoted 11 lines
xen/drivers/passthrough/arm/iommu.c +11
-0+while ( !dt_parse_phandle_with_args (np,"iommus" ,"#iommu-cells" , + index, &iommu_spec) ) + { +if ( !dt_device_is_available (iommu_spec.np) ) +break ; + + rc =iommu_fwspec_init (dev, &iommu_spec.np->dev); +if ( rc ) +break ; + + rc =iommu_fwspec_add_ids (dev, iommu_spec.args,1 );
Here you assume that there will at least always be one cells and the first cell is the IDs. For a first, #iommu-cells may be 0 (and therefore no cells) when the master IOMMU device cannot be configured.
Furthermore, the content of the #iommu-cells depends on the driver. This is why Linux provides a callback of_xlate to let the driver decide how to interpret it.
For instance, the SMMU can support either 1 or 2 cells. It also may need to look-up for other properties in the node (e.g stream-match-mask).
So I think we also want to provide the of_xlate in Xen.
Quoted 25 lines
xen/drivers/passthrough/arm/iommu.c +19
-0+if ( rc ) +break ; + + index++; + } + +/* +* Add DT device to the IOMMU if latter is present and available. +* The IOMMU driver's responsibility is to check whether dev->iommu_fwspec +* field is initialized and mark that device as protected. +*/ +if ( !rc ) + rc = ops->add_device (0 , dev); + +if ( rc <0 ) +iommu_fwspec_free (dev); + +return rc <0 ? rc :0 ; +}xen/include/asm-arm/iommu.h +0
-0
xen/include/asm-arm/iommu.h struct arch_iommu · line 36
Quoted 9 lines
xen/include/asm-arm/iommu.h +3
-0@@ -28,6 +28,9 @@ struct arch_iommuconst struct iommu_ops *iommu_get_ops (void );void iommu_set_ops (const struct iommu_ops *ops); +/* helper to add DT device to the IOMMU */ +int iommu_add_dt_device (struct dt_device_node *np); +/* mapping helpers */ int __must_check arm_iommu_map_page(struct domain *d, dfn_t dfn, mfn_t mfn, unsigned int flags,
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 5/6] iommu/arm: Introduce iommu_add_dt_device API
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi Julien
Quoted 18 lines · Oleksandr Tyshchenko
Quoted 16 lines
From: Oleksandr Tyshchenko <[email protected]>
This patch adds new
iommu_add_dt_deviceAPI for adding DT device to the IOMMU using generic IOMMU DT binding [1] and previously added "iommu_fwspec" support.New function parses the DT binding, prepares "
dev->iommu_fwspec" with correct information and calls the IOMMU driver using "add_device" callback to register new DT device.
The IOMMU driver's responsibility is to check whether "dev->iommu_fwspec" is initialized and mark that device as protected.The additional benefit here is to avoid to go through the whole DT multiple times in IOMMU driver trying to locate master devices which belong to each IOMMU device being probed.
Quoted 8 lines
Quoted 4 lines
The upcoming IPMMU driver will have "
add_device" callback implemented.I hope, this patch won't break SMMU driver's functionality, which doesn't have this callback implemented.
The last two sentence does not really belong to the commit message. So I think they should go after ---.
Anyway, the only concern for the SMMU is to not break the old bindings. New bindings are not supported, so it does not matter whether they are broken or not. Once this series is merged, we can have a look how new bindings for the SMMU can be supported.
Sounds reasonable.
Quoted 24 lines
Quoted 17 lines
[1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt
Signed-off-by Oleksandr Tyshchenko <[email protected]>---
xen/arch/arm/domain_build.c | 12 ++++++++++xen/drivers/passthrough/arm/iommu.c| 45
+++++++++++++++++++++++++++++++++++++
xen/include/asm-arm/iommu.h | 3 +++3 files changed, 60 insertions(+)xen/arch/arm/domain_build.c +0
-0xen/arch/arm/domain_build.c static int __init handle_device(struct domain · line 1244
Quoted 4 lines
xen/arch/arm/domain_build.c +0
-0@@ -1241,6 +1241,18 @@ static int __init handle_device(struct domain *d, struct dt_device_node *dev, u64 addr, size; bool need_mapping = !dt_device_for_passthrough(dev); + if ( dt_parse_phandle(dev, "iommus", 0) )I don't particularly like this check.
dt_parse_phandleis non-trivial to execute.
Quoted 1 line
TBH, what we should do is trying to call
iommu_add_dt_deviceif IOMMU is enabled. We can then return a recognizable value to tell the device is not protected.
Well. Don't really mind.
Quoted 36 lines
Quoted 23 lines
xen/arch/arm/domain_build.c +11
-0+ { + dt_dprintk("%s add to iommu\n", dt_node_full_name(dev)); + res = iommu_add_dt_device(dev); + if ( res ) + { + printk(XENLOG_ERR "Failed to add %s to the IOMMU\n", + dt_node_full_name(dev)); + return res; + } + } + nirq = dt_number_of_irq(dev); naddr = dt_number_of_address(dev);diff --git
a/xen/drivers/passthrough/arm/iommu.cb/xen/drivers/passthrough/arm/iommu.cindex 3195919..19516af 100644xen/drivers/passthrough/arm/iommu.c +0
-0@@ -113,3 +113,48 @@ int arch_iommu_populate_page_table(struct domain *d) void __hwdom_init arch_iommu_hwdom_init(struct domain *d) {}
xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 125
Quoted 10 lines
xen/drivers/passthrough/arm/iommu.c +10
-0+ +int __init iommu_add_dt_device (struct dt_device_node *np) +{ + const struct iommu_ops *ops = iommu_get_ops(); + struct dt_phandle_args iommu_spec; + struct device *dev = dt_to_dev(np); + int rc = 1, index = 0; + + if ( !iommu_enabled || !ops || !ops->add_device ) + return 0;While I agree that !iommu_enabled should return 0, for the two others I am not entirely sure this is the right thing to do.
Quoted 1 line
!ops is definitely an error because if you have the IOMMU enabled then you should have ops installed.
Agree.
Quoted 3 lines
!ops->add_device means that you will not be able to add any device using the new bindings. IOW, the device will be unusable later one as most likely the IOMMU was configured to deny any transaction. Therefore, this should return an error.
The initial reason *was* to not break SMMU which hasn't had add_device callback implemented yet. But, I got your point regarding SMMU above (the only concern for the SMMU is to not break the old bindings), so agree here.
Quoted 9 lines
Quoted 7 lines
xen/drivers/passthrough/arm/iommu.c +5
-0+ + if ( dev_iommu_fwspec_get(dev) ) + return -EEXIST; + + /* According to the
Documentation/devicetree/bindings/iommu/iommu.txt*/This file does not exist in Xen, so Linux should at least be mentioned in the comment.
Will do.
Quoted 15 lines
xen/drivers/passthrough/arm/iommu.c int arch_iommu_populate_page_table(struct domain *d) · line 141
Quoted 12 lines
+ while ( !dt_parse_phandle_with_args(np, "iommus", "#iommu-cells", + index,
&iommu_spec) ) + {
+ if ( !dt_device_is_available(iommu_spec.np) )
+ break;xen/drivers/passthrough/arm/iommu.c +6
-0+ + rc = iommu_fwspec_init(dev, &iommu_spec.np->dev); + if ( rc ) + break; + + rc = iommu_fwspec_add_ids(dev, iommu_spec.args, 1);Here you assume that there will at least always be one cells and the first cell is the IDs. For a first, #iommu-cells may be 0 (and therefore no cells) when the master IOMMU device cannot be configured.
Quoted 5 lines
Furthermore, the content of the #iommu-cells depends on the driver. This is why Linux provides a callback
of_xlateto let the driver decide how to interpret it.For instance, the SMMU can support either 1 or 2 cells. It also may need to look-up for other properties in the node (e.g stream-match-mask).
So I think we also want to provide the
of_xlatein Xen.
Hmm, I was thinking how to end up with only one callback re-used (add_device), really didn't want to add a new one (of_xlate). But, I didn't take into the account that this stuff is a really driver-depended. So, likely yes, we need to provide of_xlate callback.
May I ask some questions to clarify:
1. Do you want me to introduce of_xlate callback in a separate patch (under CONFIG_ARM?)?
2. Can we avoid introducing new API for that callback? iommu_add_dt_device will be able to call it directly.
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 5/6] iommu/arm: Introduce iommu_add_dt_device API
Hi,
Quoted 8 lines · Oleksandr
Quoted 1 line · Julien Grall
On 8/2/19 5:39 PM, Oleksandr Tyshchenko wrote:
Hmm, I was thinking how to end up with only one callback re-used (
add_device), really didn't want to add a new one (of_xlate). But, I didn't take into the account that this stuff is a really driver-depended. So, likely yes, we need to provideof_xlatecallback.May I ask some questions to clarify:
1. Do you want me to introduce
of_xlatecallback in a separate patch (underCONFIG_ARM?)?
Preferably yes. I think this wants to be under CONFIG_HAS_DEVICE_TREE rather than CONFIG_ARM.
Quoted 1 line
2. Can we avoid introducing new API for that callback?
Do you mean a wrapper for the callback? If so, yes.
Quoted 1 line
iommu_add_dt_devicewill be able to call it directly.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 6/6 iommu/arm: Add Renesas IPMMU-VMSA support
In reply to Oleksandr TyshchenkoFrom: Oleksandr Tyshchenko <[email protected]>
The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.
Please note, current driver is supposed to work only with newest Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).
The major differences compare to the Linux driver are:
1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.
So, Xen driver is supposed to be used with newest Gen3 SoC revisions only (H3 ES3.0, M3 ES3.0, etc.) which IPMMU H/W supports stage 2 translation table format.
2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address.
3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank.
---
Changes V1 -> V2:
- rewrited driver to use iommu_fwspec
- removed DT parsing code for micro-TLBs
- removed struct ipmmu_vmsa_master_cfg, dev_archdata macro
- added ipmmu_find_mmu_by_dev(), various helpers to access fwspec->iommu_priv
- implemented new callback "add_device" to add master device to IPMMU
- removed ipmmu_protect_masters()
- removed code to locate Root device in the first place, used EAGAIN to request deferred probing
- used printk_once for the system wide error messages in ipmmu_init() which don't need to be shown for every device being probed
- removed map_page/unmap_page implementation, reused them from iommu_helpers.c
- used %pd for printing domaid id
- performed various cosmetic fixes
- changed u32 -> uint32_t, u64 -> uint64_t,
unsigned int -> uint32_t where needed
- clarified TODOs
- clafiried supported SoC versions in config IPMMU_VMSA, set default to "n"
- updated comments in code, provided more accurate description, added new comments where needed
- updated patch description by providing differences between Linux/Xen implementations
- removed fields for cache snoop transaction when configuring IMTTBCR (update from Renesas BSP)
--- xen/arch/arm/platforms/Kconfig | 1 + xen/drivers/passthrough/Kconfig | 13 + xen/drivers/passthrough/arm/Makefile | 1 + xen/drivers/passthrough/arm/ipmmu-vmsa.c | 1342 ++++++++++++++++++++++++++++++ 4 files changed, 1357 insertions(+) create mode 100644 xen/drivers/passthrough/arm/ipmmu-vmsa.c
xen/arch/arm/platforms/Kconfig +1 -0
@@ -25,6 +25,7 @@ config RCAR3 bool "Renesas RCar3 support" depends on ARM_64 select HAS_SCIF + select IPMMU_VMSA ---help--- Enable all the required drivers for Renesas RCar3
xen/drivers/passthrough/Kconfig +13 -0
@@ -12,4 +12,17 @@ config ARM_SMMU Say Y here if your SoC includes an IOMMU device implementing the ARM SMMU architecture. + +config IPMMU_VMSA + bool "Renesas IPMMU-VMSA found in R-Car Gen3 SoCs" + default n + depends on ARM_64 + ---help--- + Support for implementations of the Renesas IPMMU-VMSA found + in R-Car Gen3 SoCs. + + Say Y here if you are using newest R-Car Gen3 SoCs revisions + (H3 ES3.0, M3 ES3.0, etc) which IPMMU hardware supports stage 2 + translation table format and is able to use CPU's P2M table as is. + endif
xen/drivers/passthrough/arm/Makefile +1 -0
@@ -1,2 +1,3 @@ obj-y += iommu.o iommu_helpers.o iommu_fwspec.o obj-$(CONFIG_ARM_SMMU) += smmu.o +obj-$(CONFIG_IPMMU_VMSA) += ipmmu-vmsa.o
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +1342 -0
1,342 lines not on this page · raw message
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Hi Oleksandr-san,
I can access the datasheet of this hardware, so that I reviewed this patch. I'm not familar about Xen development rulus, so that some comments might be not good fit. If so, please ignore :)
Quoted 10 lines
From: Oleksandr Tyshchenko, Sent: Saturday, August 3, 2019 1:40 AM
From: Oleksandr Tyshchenko <[email protected]>
The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.
Please note, current driver is supposed to work only with newest Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation
This should be "R-Car Gen3 SoCs", instead of "Gen3 SoCs".
Quoted 5 lines
table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).
The major differences compare to the Linux driver are:
Quoted 7 lines
1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.
So, Xen driver is supposed to be used with newest Gen3 SoC revisions only
Same here.
Quoted 1 line
(H3 ES3.0, M3 ES3.0, etc.) which IPMMU H/W supports stage 2 translation
According to the latest manual, M3 ES3.0 is named as "M3-W+".
Quoted 1 line
table format.
<snip>
Quoted 11 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0@@ -0,0 +1,1342 @@ +/* +* xen/drivers/passthrough/arm/ipmmu-vmsa.c +* +* Driver for the Renesas IPMMU-VMSA found in R-Car Gen3 SoCs. +*
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 15
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+* The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) +* which provides address translation and access protection functionalities +* to processing units and interconnect networks. +* +* Please note, current driver is supposed to work only with newest Gen3 SoCs +* revisions which IPMMU hardware supports stage 2 translation table format and +* is able to use CPU's P2M table as is. +* +* Based on Linux's IPMMU-VMSA driver from Renesas BSP: +* drivers/iommu/ipmmu-vmsa.c
So, I think the Linux's Copyrights should be described here.
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+* you can found at: +* url: git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git +* branch: v4.14.75-ltsi/rcar-3.9.6 +* commit: e206eb5b81a60e64c35fbc3a999b1a0db2b98044 +* and Xen's SMMU driver: +* xen/drivers/passthrough/arm/smmu.c +* +* Copyright (C) 2016-2019 EPAM Systems Inc. +* +* This program is free software; you can redistribute it and/or
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 35
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+* modify it under the terms and conditions of the GNU General Public +* License, version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public +* License along with this program; If not, see <http://www.gnu.org/licenses/>.
I don't know that Xen license description rule, but since a few source files have SPDX-License-Identifier, can we also use it on the driver?
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 42
Quoted 7 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7
-0+*/ + +#include <xen/delay.h> +#include <xen/err.h> +#include <xen/irq.h> +#include <xen/lib.h> +#include <xen/list.h>
I don't know that Xen passthrough driver rule though, doesn't here need #include <xen/iommu.h>? (The xen/sched.h seems to have it so that no compile error happens though.)
<snip>
Quoted 33 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +33
-0+/* Registers Definition */ +#define IM_CTX_SIZE 0x40 + +#define IMCTR 0x0000 +/* +* These fields are implemented in IPMMU-MM only. So, can be set for +* Root IPMMU only. +*/ +#define IMCTR_VA64 (1 << 29) +#define IMCTR_TRE (1 << 17) +#define IMCTR_AFE (1 << 16) +#define IMCTR_RTSEL_MASK (3 << 4) +#define IMCTR_RTSEL_SHIFT 4 +#define IMCTR_TREN (1 << 3) +/* +* These fields are common for all IPMMU devices. So, can be set for +* Cache IPMMUs as well. +*/ +#define IMCTR_INTEN (1 << 2) +#define IMCTR_FLUSH (1 << 1) +#define IMCTR_MMUEN (1 << 0) +#define IMCTR_COMMON_MASK (7 << 0) + +#define IMCAAR 0x0004 + +#define IMTTBCR 0x0008 +#define IMTTBCR_EAE (1 << 31) +#define IMTTBCR_PMB (1 << 30) +#define IMTTBCR_SH1_NON_SHAREABLE (0 << 28) +#define IMTTBCR_SH1_OUTER_SHAREABLE (2 << 28) +#define IMTTBCR_SH1_INNER_SHAREABLE (3 << 28) +#define IMTTBCR_SH1_MASK (3 << 28) +#define IMTTBCR_ORGN1_NC (0 << 26)
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 201
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+#define IMTTBCR_ORGN1_WB_WA (1 << 26) +#define IMTTBCR_ORGN1_WT (2 << 26) +#define IMTTBCR_ORGN1_WB (3 << 26) +#define IMTTBCR_ORGN1_MASK (3 << 26) +#define IMTTBCR_IRGN1_NC (0 << 24) +#define IMTTBCR_IRGN1_WB_WA (1 << 24) +#define IMTTBCR_IRGN1_WT (2 << 24) +#define IMTTBCR_IRGN1_WB (3 << 24) +#define IMTTBCR_IRGN1_MASK (3 << 24) +#define IMTTBCR_TSZ1_MASK (1f << 16)
At the moment, no one uses it though, this should be (0x1f << 16).
<snip>
+/* Xen IOMMU ops */
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 811
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8
-0+static int __must_check ipmmu_iotlb_flush_all (struct domain *d) +{ +struct ipmmu_vmsa_xen_domain *xen_domain =dom_iommu (d)->arch.priv; + +if ( !xen_domain || !xen_domain->root_domain ) +return 0 ; + +spin_lock (&xen_domain->lock);
Is local irq is already disabled here?
If no, you should use spin_lock_irqsave() because the ipmmu_irq() also gets the lock.
# To be honest, in normal case, any irq on the current implementation
# should not happen though.
Quoted 6 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+ipmmu_tlb_invalidate (xen_domain->root_domain); +spin_unlock (&xen_domain->lock); + +return 0 ; +} +
<snip>
Quoted 4 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +4
-0+static int ipmmu_assign_device (struct domain *d,u8 devfn,struct device *dev, +uint32_t flag) +{ +struct ipmmu_vmsa_xen_domain *xen_domain =dom_iommu (d)->arch.priv;
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 936
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+struct ipmmu_vmsa_domain *domain; +int ret; + +if ( !xen_domain ) +return -EINVAL ; + +if ( !to_ipmmu (dev) ) +return -ENODEV ; + +spin_lock (&xen_domain->lock);
Same here.
Quoted 71 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +71
-0+/* +* The IPMMU context for the Xen domain is not allocated beforehand +* (at the Xen domain creation time), but on demand only, when the first +* master device being attached to it. +* Create Root IPMMU domain which context will be mapped to this Xen domain +* if not exits yet. +*/ +if ( !xen_domain->root_domain ) + { + domain =ipmmu_alloc_root_domain (d); +if (IS_ERR (domain) ) + { + ret =PTR_ERR (domain); + goto out; + } + + xen_domain->root_domain = domain; + } + +if (to_domain (dev) ) + { +dev_err (dev,"Already attached to IPMMU domain\n" ); + ret = -EEXIST ; + goto out; + } + +/* +* Master devices behind the same Cache IPMMU can be attached to the same +* Cache IPMMU domain. +* Before creating new IPMMU domain check to see if the required one +* already exists for this Xen domain. +*/ + domain =ipmmu_get_cache_domain (d, dev); +if ( !domain ) + { +/* Create new IPMMU domain this master device will be attached to. */ + domain =ipmmu_alloc_cache_domain (d); +if (IS_ERR (domain) ) + { + ret =PTR_ERR (domain); + goto out; + } + +/* Chain new IPMMU domain to the Xen domain. */ +list_add (&domain->list, &xen_domain->cache_domains); + } + + ret =ipmmu_attach_device (domain, dev); +if ( ret ) + { +/* +* Destroy Cache IPMMU domain only if there are no master devices +* attached to it. +*/ +if ( !domain->refcount ) +ipmmu_free_cache_domain (domain); + } +else + { + domain->refcount++; +set_domain (dev, domain); + } + +out: +spin_unlock (&xen_domain->lock); + +return ret; +} + +static int ipmmu_deassign_device (struct domain *d,struct device *dev) +{
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1018
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+struct ipmmu_vmsa_xen_domain *xen_domain =dom_iommu (d)->arch.priv; +struct ipmmu_vmsa_domain *domain =to_domain (dev); + +if ( !domain || domain->d != d ) + { +dev_err (dev,"Not attached to %pd\n" , d); +return -ESRCH ; + } + +spin_lock (&xen_domain->lock);
Same here.
Quoted 15 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +15
-0+ipmmu_detach_device (domain, dev); +set_domain (dev,NULL ); + domain->refcount--; + +/* +* Destroy Cache IPMMU domain only if there are no master devices +* attached to it. +*/ +if ( !domain->refcount ) +ipmmu_free_cache_domain (domain); + +spin_unlock (&xen_domain->lock); + +return 0 ; +}
<snip>
Quoted 11 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +11
-0+static void __hwdom_init ipmmu_iommu_hwdom_init(struct domain *d) +{ +/* Set to false options not supported on ARM. */ +if ( iommu_hwdom_inclusive ) +printk (XENLOG_WARNING "ipmmu: map-inclusive dom0-iommu option is not supported on ARM\n" ); + iommu_hwdom_inclusive = false; +if ( iommu_hwdom_reserved ==1 ) +printk (XENLOG_WARNING "ipmmu: map-reserved dom0-iommu option is not supported on ARM\n" ); + iommu_hwdom_reserved =0 ; + +arch_iommu_hwdom_init (d);
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1154
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+} + +static void ipmmu_iommu_domain_teardown (struct domain *d) +{ +struct ipmmu_vmsa_xen_domain *xen_domain =dom_iommu (d)->arch.priv; + +if ( !xen_domain ) +return ; + +spin_lock (&xen_domain->lock);
Same here.
Quoted 5 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0+/* +* Destroy Root IPMMU domain which context is mapped to this Xen domain +* if exits. +*/ +if ( xen_domain->root_domain )
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1169
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ipmmu_free_root_domain (xen_domain->root_domain); + +spin_unlock (&xen_domain->lock); + +/* +* We assume that all master devices have already been detached from +* this Xen domain and there must be no associated Cache IPMMU domains +* in use. +*/ +ASSERT (list_empty (&xen_domain->cache_domains));
I think this should be in the spin lock held by &xen_domain->lock.
Quoted 9 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +9
-0+xfree (xen_domain); +dom_iommu (d)->arch.priv =NULL ; +} + +static const struct iommu_ops ipmmu_iommu_ops = +{ + .init = ipmmu_iommu_domain_init, + .hwdom_init = ipmmu_iommu_hwdom_init, + .teardown = ipmmu_iommu_domain_teardown,
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1188
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+ .iotlb_flush = ipmmu_iotlb_flush, + .iotlb_flush_all = ipmmu_iotlb_flush_all, + .assign_device = ipmmu_assign_device, + .reassign_device = ipmmu_reassign_device, + .map_page = arm_iommu_map_page, + .unmap_page = arm_iommu_unmap_page, + .add_device = ipmmu_add_device, +}; + +/* RCAR GEN3 product and cut information. */
"R-Car Gen3 SoCs" is better than "RCAR GEN3".
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1191
Quoted 3 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +3
-0+#define RCAR_PRODUCT_MASK 0x00007F00 +#define RCAR_PRODUCT_H3 0x00004F00 +#define RCAR_PRODUCT_M3 0x00005200
At least, I think we should be M3W, instead of M3. # FYI, M3-W and M3-W+ are the same value.
<snip>
Best regards,
Yoshihiro Shimoda
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Hi, Shimoda-san.
Thank you for the review.
Quoted 5 lines
Quoted 5 lines
From: Oleksandr Tyshchenko, Sent: Saturday, August 3, 2019 1:40 AM
From: Oleksandr Tyshchenko <[email protected]>
Quoted 7 lines
Quoted 6 lines
The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks.
Please note, current driver is supposed to work only with newest Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation
This should be "R-Car Gen3 SoCs", instead of "Gen3 SoCs".
Will update.
Quoted 6 lines
Quoted 6 lines
table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA).
The major differences compare to the Linux driver are:
Quoted 8 lines
Quoted 7 lines
1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver.
So, Xen driver is supposed to be used with newest Gen3 SoC revisions only
Same here.
ok
Quoted 3 lines
Quoted 2 lines
(H3 ES3.0, M3 ES3.0, etc.) which IPMMU H/W supports stage 2 translation
According to the latest manual, M3 ES3.0 is named as "M3-W+".
Will update.
Quoted 14 lines
Quoted 14 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8
-0@@ -0,0 +1,1342 @@ +/* +* xen/drivers/passthrough/arm/ipmmu-vmsa.c +* +* Driver for the Renesas IPMMU-VMSA found in R-Car Gen3 SoCs. +* +* The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) +* which provides address translation and access protection functionalities +* to processing units and interconnect networks.
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 15
Quoted 7 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7
-0+* +* Please note, current driver is supposed to work only with newest Gen3 SoCs +* revisions which IPMMU hardware supports stage 2 translation table format and +* is able to use CPU's P2M table as is. +* +* Based on Linux's IPMMU-VMSA driver from Renesas BSP: +* drivers/iommu/ipmmu-vmsa.c So, I think the Linux's Copyrights should be described here.
Yes, will add.
Quoted 21 lines
Quoted 11 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+* you can found at: +* url: git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git +* branch: v4.14.75-ltsi/rcar-3.9.6 +* commit: e206eb5b81a60e64c35fbc3a999b1a0db2b98044 +* and Xen's SMMU driver: +* xen/drivers/passthrough/arm/smmu.c +* +* Copyright (C) 2016-2019 EPAM Systems Inc. +* +* This program is free software; you can redistribute it and/or xen/drivers/passthrough/arm/ipmmu-vmsa.c line 35
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+* modify it under the terms and conditions of the GNU General Public +* License, version 2, as published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +* General Public License for more details. +* +* You should have received a copy of the GNU General Public +* License along with this program; If not, see <http://www.gnu.org/licenses/>.
Quoted 2 lines
I don't know that Xen license description rule, but since a few source files have SPDX-License-Identifier, can we also use it on the driver?
I am afraid, I don't know a correct answer for this question. I would leave this to maintainers.
I just followed sample copyright notice for GPL v2 License according to the document:
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=CONTRIBUTING
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 42
Quoted 7 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7
-0+*/ + +#include <xen/delay.h> +#include <xen/err.h> +#include <xen/irq.h> +#include <xen/lib.h> +#include <xen/list.h> I don't know that Xen passthrough driver rule though, doesn't here need #include <
xen/iommu.h>? (Thexen/sched.hseems to have it so that no compile error happens though.)
Probably, yes, I should have included that header.
Quoted 12 lines
Quoted 12 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +12
-0+ +#define IMTTBCR 0x0008 +#define IMTTBCR_EAE (1 << 31) +#define IMTTBCR_PMB (1 << 30) +#define IMTTBCR_SH1_NON_SHAREABLE (0 << 28) +#define IMTTBCR_SH1_OUTER_SHAREABLE (2 << 28) +#define IMTTBCR_SH1_INNER_SHAREABLE (3 << 28) +#define IMTTBCR_SH1_MASK (3 << 28) +#define IMTTBCR_ORGN1_NC (0 << 26) +#define IMTTBCR_ORGN1_WB_WA (1 << 26) +#define IMTTBCR_ORGN1_WT (2 << 26) +#define IMTTBCR_ORGN1_WB (3 << 26)
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 201
Quoted 7 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7
-0+#define IMTTBCR_ORGN1_MASK (3 << 26) +#define IMTTBCR_IRGN1_NC (0 << 24) +#define IMTTBCR_IRGN1_WB_WA (1 << 24) +#define IMTTBCR_IRGN1_WT (2 << 24) +#define IMTTBCR_IRGN1_WB (3 << 24) +#define IMTTBCR_IRGN1_MASK (3 << 24) +#define IMTTBCR_TSZ1_MASK (1f << 16)At the moment, no one uses it though, this should be (
0x1f<< 16).
Will correct.
Quoted 14 lines
<snip>
+/* Xen IOMMU ops */xen/drivers/passthrough/arm/ipmmu-vmsa.c line 811
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8
-0+static int __must_check ipmmu_iotlb_flush_all (struct domain *d) +{ +struct ipmmu_vmsa_xen_domain *xen_domain =dom_iommu (d)->arch.priv; + +if ( !xen_domain || !xen_domain->root_domain ) +return 0 ; + +spin_lock (&xen_domain->lock);Is local irq is already disabled here?
If no, you should usespin_lock_irqsave()because theipmmu_irq()also gets the lock.
No, it is not disabled. But, ipmmu_irq() uses another mmu->lock. So, I think, there won't be a deadlock.
Or I really missed something?
If we worry about ipmmu_tlb_invalidate() which is called here (to perform a flush by request from P2M code, which manages a page table) and from the irq handler (to perform a flush to resume address translation), I could use a tasklet to schedule ipmmu_tlb_invalidate() from the irq handler then. This way we would get this serialized. What do you think?
Quoted 2 lines
# To be honest, in normal case, any irq on the current implementation
# should not happen though.
Agree here.
Quoted 8 lines
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8
-0+/* +* Destroy Root IPMMU domain which context is mapped to this Xen domain +* if exits. +*/ +if ( xen_domain->root_domain ) +ipmmu_free_root_domain (xen_domain->root_domain); + +spin_unlock (&xen_domain->lock);
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1169
Quoted 7 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7
-0+ +/* +* We assume that all master devices have already been detached from +* this Xen domain and there must be no associated Cache IPMMU domains +* in use. +*/ +ASSERT (list_empty (&xen_domain->cache_domains));I think this should be in the spin lock held by
&xen_domain->lock.
OK. Will put spin_unlock after it.
Quoted 13 lines
Quoted 13 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +12
-0+xfree (xen_domain); +dom_iommu (d)->arch.priv =NULL ; +} + +static const struct iommu_ops ipmmu_iommu_ops = +{ + .init = ipmmu_iommu_domain_init, + .hwdom_init = ipmmu_iommu_hwdom_init, + .teardown = ipmmu_iommu_domain_teardown, + .iotlb_flush = ipmmu_iotlb_flush, + .iotlb_flush_all = ipmmu_iotlb_flush_all, + .assign_device = ipmmu_assign_device,
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1188
Quoted 7 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +7
-0+ .reassign_device = ipmmu_reassign_device, + .map_page = arm_iommu_map_page, + .unmap_page = arm_iommu_unmap_page, + .add_device = ipmmu_add_device, +}; + +/* RCAR GEN3 product and cut information. */ "R-Car Gen3 SoCs" is better than "RCAR GEN3".
Will update.
Quoted 6 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1191
Quoted 4 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +3
-0+#define RCAR_PRODUCT_MASK 0x00007F00 +#define RCAR_PRODUCT_H3 0x00004F00 +#define RCAR_PRODUCT_M3 0x00005200At least, I think we should be M3W, instead of M3. # FYI, M3-W and M3-W+ are the same value.
Will update.
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
(+ Lars)
Hi,
Quoted 22 lines · Oleksandr
Quoted 22 lines
Quoted 22 lines
+ * you can found at:
+ * url:git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git
+ * branch: v4.14.75-ltsi/rcar-3.9.6
+ * commit:e206eb5b81a60e64c35fbc3a999b1a0db2b98044
+ * and Xen's SMMU driver:
+ *xen/drivers/passthrough/arm/smmu.c
+ *
+ * Copyright (C) 2016-2019 EPAM Systems Inc.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; If not, see
<http://www.gnu.org/licenses/>.
Quoted 8 lines
Quoted 2 lines
I don't know that Xen license description rule, but since a few source files have SPDX-License-Identifier, can we also use it on the driver?
I am afraid, I don't know a correct answer for this question. I would leave this to maintainers.
I just followed sample copyright notice for GPL v2 License according to the document:
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=CONTRIBUTING
The file CONTRIBUTING is only giving example of common example of license. So I think this is fine to use SPDX, the more they are already used. The only request is to put either SDPX or the full-blown text but not the two :). Lars, any objection?
I am quite in favor of SPDX because it is easier to find out the license. With the full-blown text, the text may slightly vary between licenses. For instance, the only difference between GPLv2 and GPLv2+ is ",or (at your option) any later version". I let you imagine how it can be easy to miss it when reviewing ;).
We had a discussion last year about using SPDX in Xen code base but I never got the time to formally suggest it.
Quoted 12 lines
Quoted 11 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 42
Quoted 8 lines
+ */
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +6
-0+ +#include <xen/delay.h> +#include <xen/err.h> +#include <xen/irq.h> +#include <xen/lib.h> +#include <xen/list.h> I don't know that Xen passthrough driver rule though, doesn't here need #include <
xen/iommu.h>? (Thexen/sched.hseems to have it so that no compile error happens though.)
Quoted 1 line
Probably, yes, I should have included that header.
I am fine either way :). The indirect inclusion happens quite often and we only notice it when someone decide to rework the headers.
[...]
Quoted 14 lines
Quoted 12 lines
+/* Xen IOMMU ops */
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 811
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8
-0+static int __must_check ipmmu_iotlb_flush_all (struct domain *d) +{ + struct ipmmu_vmsa_xen_domain *xen_domain = dom_iommu(d)->arch.priv; + + if ( !xen_domain || !xen_domain->root_domain ) + return 0; + + spin_lock(&xen_domain->lock);Is local irq is already disabled here?
If no, you should usespin_lock_irqsave()because theipmmu_irq()also gets the lock.
Quoted 5 lines
No, it is not disabled. But,
ipmmu_irq()uses anothermmu->lock. So, I think, there won't be a deadlock.Or I really missed something?
If we worry about
ipmmu_tlb_invalidate()which is called here (to perform a flush by request from P2M code, which manages a page table) and from the irq handler (to perform a flush to resume address translation), I could use a tasklet to scheduleipmmu_tlb_invalidate()from the irq handler then. This way we would get this serialized. What do you think?
I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.
This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).
[...]
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
+ /*
+ * Destroy Root IPMMU domain which context is mapped to this Xen domain
+ * if exits.
+ */
+ if (xen_domain->root_domain)
+ipmmu_free_root_domain(xen_domain->root_domain);xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +4
-0+ + spin_unlock(&xen_domain->lock); + + /*Quoted 8 lines
Quoted 7 lines
+ * We assume that all master devices have already been detached from
+ * this Xen domain and there must be no associated Cache IPMMU domains
+ * in use.
+ */
+ ASSERT(list_empty(&xen_domain->cache_domains));I think this should be in the spin lock held by
&xen_domain->lock.
Quoted 1 line
OK. Will put
spin_unlockafter it.
The spin_lock is actually pointless here. This is done when the domain is destroyed, so nobody should touch it.
If you think concurrent access can still happen, then you are going to be in deep trouble as you free the xen_domain (and therefore the spinlock) below :).
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Hi, Julien.
Sorry for the possible format issues.
Quoted 15 lines
Quoted 11 lines
No, it is not disabled. But,
ipmmu_irq()uses anothermmu->lock. So, I think, there won't be a deadlock.Or I really missed something?
If we worry about
ipmmu_tlb_invalidate()which is called here (to perform a flush by request from P2M code, which manages a page table) and from the irq handler (to perform a flush to resume address translation), I could use a tasklet to scheduleipmmu_tlb_invalidate()from the irq handler then. This way we would get this serialized. What do you think?I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.
Quoted 9 lines
This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).
I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:
Sorry if I was unclear.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Quoted 1 line · Oleksandr Tyshchenko
Hi, Julien.
Hi,
Quoted 27 lines
Sorry for the possible format issues.
> No, it is not disabled. But,
ipmmu_irq()uses anothermmu->lock. So, I > think, there won't be a deadlock.
>
> Or I really missed something?
>
> If we worry aboutipmmu_tlb_invalidate()which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to scheduleipmmu_tlb_invalidate()> from the irq handler then. This way we would get this serialized. What > do you think?I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.
This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).
Quoted 5 lines
I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:
Sorry if I was unclear.
My mistake, I misread what you wrote.
I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?
Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Quoted 3 lines
Hi,
Hi, Julien.
Quoted 33 lines
Quoted 33 lines
Sorry for the possible format issues.
> No, it is not disabled. But,
ipmmu_irq()uses anothermmu->lock. So, I > think, there won't be a deadlock.
>
> Or I really missed something?
>
> If we worry aboutipmmu_tlb_invalidate()which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to scheduleipmmu_tlb_invalidate()> from the irq handler then. This way we would get this serialized. What > do you think?I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.
This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).
I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:
Quoted 7 lines
Quoted 1 line
Sorry if I was unclear.
My mistake, I misread what you wrote.
I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?
Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?
Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.
So, my worry about calling ipmmu_tlb_invalidate() directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.
But, with one remark, as you have already noted, we are not trying to handle/fix this fault (update page table entries), driver doesn't manage page table and is not aware what the page table is. What is more, it is unclear what actually need to be fixed in the page table which is a CPU page table as the same time.
I have heard there is a break-before-make sequence when updating the page table. So, if device in a domain is issuing DMA somewhere in the middle of updating a page table, theoretically, we might hit into this fault. In this case the page table is correct and we don't need to fix anything... Being honest, I have never seen a fault caused by break-before-make sequence.
Quoted 3 lines
Cheers,
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Hi,
Removing Lars there is no need to spam him with technical discussion :)
Quoted 49 lines · Oleksandr
Quoted 3 lines
Hi,
Hi, Julien.
Quoted 33 lines
Quoted 33 lines
Sorry for the possible format issues.
> No, it is not disabled. But,
ipmmu_irq()uses anothermmu->lock. So, I > think, there won't be a deadlock.
>
> Or I really missed something?
>
> If we worry aboutipmmu_tlb_invalidate()which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to scheduleipmmu_tlb_invalidate()> from the irq handler then. This way we would get this serialized. What > do you think?I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.
This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).
I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:
Quoted 7 lines
Quoted 1 line
Sorry if I was unclear.
My mistake, I misread what you wrote.
I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?
Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?
Quoted 6 lines
Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.
So, my worry about calling
ipmmu_tlb_invalidate()directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.
Well, I don't have the TRM... so my point of reference is Linux. Why does upstream not do the TLB flush?
I have been told this is an errata on the IPMMU. Is it related to the series posted on linux-iommu [1]?
Quoted 4 lines
But, with one remark, as you have already noted, we are not trying to handle/fix this fault (update page table entries), driver doesn't manage page table and is not aware what the page table is. What is more, it is unclear what actually need to be fixed in the page table which is a CPU page table as the same time.
I have heard there is a break-before-make sequence when updating the page table. So, if device in a domain is issuing DMA somewhere in the middle of updating a page table, theoretically, we might hit into this fault. In this case the page table is correct and we don't need to fix anything... Being honest, I have never seen a fault caused by break-before-make sequence.
Ok, so it looks like you are trying to fix [1]. My first concern here is there are no ground for someone without access to the TRM why this is done.
Furthermore, AFAICT, the patch series never reached upstream. So is it present on all revision of GEN3?
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Hi
Quoted 42 lines
Quoted 42 lines
Quoted 40 lines
Quoted 28 lines
Sorry for the possible format issues.
> No, it is not disabled. But,
ipmmu_irq()uses anothermmu->lock. So, I > think, there won't be a deadlock.
>
> Or I really missed something?
>
> If we worry aboutipmmu_tlb_invalidate()which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to scheduleipmmu_tlb_invalidate()> from the irq handler then. This way we would get this serialized. What > do you think?I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.
This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).
Quoted 6 lines
I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:
Sorry if I was unclear.
My mistake, I misread what you wrote.
I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?
Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?
Quoted 8 lines
Quoted 6 lines
Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.
So, my worry about calling
ipmmu_tlb_invalidate()directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.Well, I don't have the TRM... so my point of reference is Linux. Why does upstream not do the TLB flush?
I have no idea regarding that.
Quoted 3 lines
I have been told this is an errata on the IPMMU. Is it related to the series posted on linux-iommu [1]?
I don't think, the TLB flush we are speaking about, is related to that series [1] somehow. This TLB flush, I think, is just the last step in a sequence of actions which should be performed when the error occurs, no more no less. This is how I understand this.
Quoted 7 lines
Quoted 5 lines
But, with one remark, as you have already noted, we are not trying to handle/fix this fault (update page table entries), driver doesn't manage page table and is not aware what the page table is. What is more, it is unclear what actually need to be fixed in the page table which is a CPU page table as the same time.
I have heard there is a break-before-make sequence when updating the page table. So, if device in a domain is issuing DMA somewhere in the middle of updating a page table, theoretically, we might hit into this fault. In this case the page table is correct and we don't need to fix anything... Being honest, I have never seen a fault caused by break-before-make sequence.
Ok, so it looks like you are trying to fix [1]. My first concern here is there are no ground for someone without access to the TRM why this is done.
No, I am definitely not trying to fix [1]. I just follow the BSP driver I am based on, which in turn follows the TRM. I can extend a comment in the code before calling ipmmu_tlb_invalidate().
Quoted 2 lines
Furthermore, AFAICT, the patch series never reached upstream. So is it present on all revision of GEN3?
I think, that the newest SoCs revisions (ES 3.0) this driver is supposed to support only, are *not* affected by that errata. And *not* require such workaround.
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Quoted 54 lines · Oleksandr
Hi
Quoted 42 lines
Quoted 42 lines
Quoted 40 lines
Quoted 28 lines
Sorry for the possible format issues.
> No, it is not disabled. But,
ipmmu_irq()uses anothermmu->lock. So, I > think, there won't be a deadlock.
>
> Or I really missed something?
>
> If we worry aboutipmmu_tlb_invalidate()which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to scheduleipmmu_tlb_invalidate()> from the irq handler then. This way we would get this serialized. What > do you think?I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.
This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).
Quoted 6 lines
I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:
Sorry if I was unclear.
My mistake, I misread what you wrote.
I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?
Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?
Quoted 8 lines
Quoted 6 lines
Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.
So, my worry about calling
ipmmu_tlb_invalidate()directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.Well, I don't have the TRM... so my point of reference is Linux. Why does upstream not do the TLB flush?
Quoted 7 lines
I have no idea regarding that. >
Quoted 3 lines
I have been told this is an errata on the IPMMU. Is it related to the series posted on linux-iommu [1]?
I don't think, the TLB flush we are speaking about, is related to that series [1] somehow. This TLB flush, I think, is just the last step in a sequence of actions which should be performed when the error occurs, no more no less. This is how I understand this.
If you have to flush the TLBs in the IRQ context then something has gone really wrong.
I don't deny that Break-Before-Make is an issue. However, if it is handled correctly in the P2M code. You should only be there because there are no mapping in the TLBs for the address accessed. So flushing the TLBs should be unnecessary, unless your TLB is also caching invalid entry?
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
But, with one remark, as you have already noted, we are not trying to handle/fix this fault (update page table entries), driver doesn't manage page table and is not aware what the page table is. What is more, it is unclear what actually need to be fixed in the page table which is a CPU page table as the same time.
I have heard there is a break-before-make sequence when updating the page table. So, if device in a domain is issuing DMA somewhere in the middle of updating a page table, theoretically, we might hit into this fault. In this case the page table is correct and we don't need to fix anything... Being honest, I have never seen a fault caused by break-before-make sequence.
Ok, so it looks like you are trying to fix [1]. My first concern here is there are no ground for someone without access to the TRM why this is done.
No, I am definitely not trying to fix [1]. I just follow the BSP driver I am based on, which in turn follows the TRM. I can extend a comment in the code before calling
ipmmu_tlb_invalidate().
The fact that the code is in the BSP and not in Linux is worrying me. The commit message in the BSP is quite unhelpful to determine the exact reason.
It either means Linux rejected the patch or this was not submitted. Either way, this should be understood why such discrepancy.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Hi, Julien.
Quoted 55 lines
Quoted 55 lines
Quoted 42 lines
Quoted 42 lines
Quoted 40 lines
Quoted 28 lines
Sorry for the possible format issues.
> No, it is not disabled. But,
ipmmu_irq()uses anothermmu->lock. So, I > think, there won't be a deadlock.
>
> Or I really missed something?
>
> If we worry aboutipmmu_tlb_invalidate()which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to scheduleipmmu_tlb_invalidate()> from the irq handler then. This way we would get this serialized. What > do you think?I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.
This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).
Quoted 6 lines
I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:
Sorry if I was unclear.
My mistake, I misread what you wrote.
I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?
Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?
Quoted 8 lines
Quoted 6 lines
Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.
So, my worry about calling
ipmmu_tlb_invalidate()directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.Well, I don't have the TRM... so my point of reference is Linux. Why does upstream not do the TLB flush?
I have no idea regarding that. >
Quoted 2 lines
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
I have been told this is an errata on the IPMMU. Is it related to the series posted on linux-iommu [1]?
I don't think, the TLB flush we are speaking about, is related to that series [1] somehow. This TLB flush, I think, is just the last step in a sequence of actions which should be performed when the error occurs, no more no less. This is how I understand this.
If you have to flush the TLBs in the IRQ context then something has gone really wrong.
I don't deny that Break-Before-Make is an issue. However, if it is handled correctly in the P2M code. You should only be there because there are no mapping in the TLBs for the address accessed. So flushing the TLBs should be unnecessary, unless your TLB is also caching invalid entry?
Sorry, I don't quite understand why we need to worry about this flush too much for a case which won't occur in normal condition (if everything is correct). Why we can't just consider this flush as a required action, which needed to exit from the error state and resume stopped address translation request. The same required action as "clearing error status flags" before. We are not trying to understand, why is it so necessary to clear error flags when error happens, or can we end up without clearing it, for example. We just follow what described in document. The same, I think, we have for that flush, if described, then should be followed. Looks like this flush acts as a trigger to unblock stopped transaction in that particular case.
Different H/W could have different restoring sequences. Some H/W requires just clearing error status, other H/W requires full re-initialization in a specific order to recover from the error state.
Please correct me if I am wrong.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
But, with one remark, as you have already noted, we are not trying to handle/fix this fault (update page table entries), driver doesn't manage page table and is not aware what the page table is. What is more, it is unclear what actually need to be fixed in the page table which is a CPU page table as the same time.
I have heard there is a break-before-make sequence when updating the page table. So, if device in a domain is issuing DMA somewhere in the middle of updating a page table, theoretically, we might hit into this fault. In this case the page table is correct and we don't need to fix anything... Being honest, I have never seen a fault caused by break-before-make sequence.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Ok, so it looks like you are trying to fix [1]. My first concern here is there are no ground for someone without access to the TRM why this is done.
No, I am definitely not trying to fix [1]. I just follow the BSP driver I am based on, which in turn follows the TRM. I can extend a comment in the code before calling
ipmmu_tlb_invalidate().The fact that the code is in the BSP and not in Linux is worrying me. The commit message in the BSP is quite unhelpful to determine the exact reason.
It either means Linux rejected the patch or this was not submitted. Either way, this should be understood why such discrepancy.
I failed to find something similar in the ML. So, probably, was not submitted. Hope, we will be able to clarify a reason.
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Hi Oleksandr,
Quoted 57 lines · Oleksandr
Quoted 57 lines
Quoted 57 lines
Quoted 42 lines
Quoted 42 lines
Quoted 40 lines
Quoted 28 lines
Sorry for the possible format issues.
> No, it is not disabled. But,
ipmmu_irq()uses anothermmu->lock. So, I > think, there won't be a deadlock.
>
> Or I really missed something?
>
> If we worry aboutipmmu_tlb_invalidate()which is called here (to > perform a flush by request from P2M code, which manages a page table) > and from the irq handler (to perform a flush to resume address > translation), I could use a tasklet to scheduleipmmu_tlb_invalidate()> from the irq handler then. This way we would get this serialized. What > do you think?I am afraid a tasklet is not an option. You need to perform the TLB flush when requested otherwise you are introducing a security issue.
This is because as soon as a region is unmapped in the page table, we remove the drop the reference on any page backing that region. When the reference is dropped to zero, the page can be reallocated to another domain or even Xen. If the TLB flush happen after, then the guest may still be able to access the page for a short time if the translation has been cached by the any TLB (IOMMU, Processor).
Quoted 6 lines
I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:
Sorry if I was unclear.
My mistake, I misread what you wrote.
I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?
Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?
Quoted 8 lines
Quoted 6 lines
Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.
So, my worry about calling
ipmmu_tlb_invalidate()directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.Well, I don't have the TRM... so my point of reference is Linux. Why does upstream not do the TLB flush?
I have no idea regarding that. >
Quoted 3 lines
I have been told this is an errata on the IPMMU. Is it related to the series posted on linux-iommu [1]?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I don't think, the TLB flush we are speaking about, is related to that series [1] somehow. This TLB flush, I think, is just the last step in a sequence of actions which should be performed when the error occurs, no more no less. This is how I understand this.
If you have to flush the TLBs in the IRQ context then something has gone really wrong.
I don't deny that Break-Before-Make is an issue. However, if it is handled correctly in the P2M code. You should only be there because there are no mapping in the TLBs for the address accessed. So flushing the TLBs should be unnecessary, unless your TLB is also caching invalid entry?
Sorry, I don't quite understand why we need to worry about this flush too much for a case which won't occur in normal condition (if everything is correct). Why we can't just consider this flush as a required action,
A translation error can be easy to reach. For instance if the guest does not program the Device correctly and request to access an address that is not mapped.
Quoted 1 line
which needed to exit from the error state and resume stopped address translation request. The same required action as "clearing error status flags" before. We are not trying to understand, why is it so necessary to clear error flags when error happens, or can we end up without clearing it, for example. We just follow what described in document. The same, I think, we have for that flush, if described, then should be followed. Looks like this flush acts as a trigger to unblock stopped transaction in that particular case.
What will actually happen if the transaction fail again? For instance, if the IOVA was not mapped. Will you receive the interrupt again? If so, are you going to make the flush again and again until the guest is killed?
Quoted 4 lines
Different H/W could have different restoring sequences. Some H/W requires just clearing error status, other H/W requires full re-initialization in a specific order to recover from the error state.
Please correct me if I am wrong.
I am not confident to accept any code that I don't understand or I don't find sensible. As I pointed out in my previous e-mail, this hasn't reached upstream so something looks quite fishy here.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Hi Julien.
Sorry for the possible format issues.
чт, 8 авг. 2019 г., 23:32 Julien Grall <[email protected]>:
Quoted 105 lines
Hi Oleksandr,
Quoted 41 lines · Oleksandr
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Sorry for the possible format issues.
> No, it is not disabled. But,
ipmmu_irq()uses anothermmu->lock. So, I
> think, there won't be a deadlock.
>
> Or I really missed something?
>
> If we worry aboutipmmu_tlb_invalidate()which is called
here (to
> perform a flush by request from P2M code, which manages a
page table)
> and from the irq handler (to perform a flush to resume
address
> translation), I could use a tasklet to scheduleipmmu_tlb_invalidate()
> from the irq handler then. This way we would get this
serialized. What
> do you think?I am afraid a tasklet is not an option. You need to perform
the TLB
flush when requested otherwise you are introducing a security
issue.This is because as soon as a region is unmapped in the page
table, we
remove the drop the reference on any page backing that
region. When the
reference is dropped to zero, the page can be reallocated to
another
domain or even Xen. If the TLB flush happen after, then the
guest may
still be able to access the page for a short time if the
translation has
been cached by the any TLB (IOMMU, Processor).Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
I understand this. I am not proposing to delay a requested by P2M code TLB flush in any case. I just propose to issue TLB flush (which we have to perform in case of page faults, to resolve error condition and resume translations) from a tasklet rather than from interrupt handler directly. This is the TLB flush I am speaking about:
Quoted 52 lines
Quoted 51 lines
Quoted 35 lines
Quoted 28 lines
Quoted 18 lines
Quoted 6 lines
Quoted 3 lines
Sorry if I was unclear.
My mistake, I misread what you wrote.
Quoted 7 lines
I found the flush in the renesas-bsp and not Linux upstream but it is not clear why this is actually required. You are not fixing any translation error. So what this flush will do?
Regarding the placement of the flush, then if you execute in a tasklet it will likely be done later on when the IRQ has been acknowledge. What's the implication to delay it?
Looks like, there is no need to put this flush into a tasklet. As I understand from Shimoda-san's answer it is OK to call flush here.
Quoted 9 lines
So, my worry about calling
ipmmu_tlb_invalidate()directly from the interrupt handler is not actual anymore.
----------
This is my understanding regarding the flush purpose here. This code, just follows the TRM, no more no less,
which mentions about a need to flush TLB after clearing error status register and updating a page table entries (which, I assume, means to resolve a reason why translation/page fault error actually have happened) to resume address translation request.Quoted 2 lines
Well, I don't have the TRM... so my point of reference is Linux. Why does upstream not do the TLB flush?
I have no idea regarding that. >
Quoted 2 lines
Quoted 7 lines
Quoted 2 lines
I have been told this is an errata on the IPMMU. Is it related to the series posted on linux-iommu [1]?
I don't think, the TLB flush we are speaking about, is related to that series [1] somehow. This TLB flush, I think, is just the last step in a sequence of actions which should be performed when the error occurs, no more no less. This is how I understand this.
If you have to flush the TLBs in the IRQ context then something has gone really wrong.
I don't deny that Break-Before-Make is an issue. However, if it is handled correctly in the P2M code. You should only be there because there are no mapping in the TLBs for the address accessed. So flushing the TLBs should be unnecessary, unless your TLB is also caching invalid entry?
Quoted 8 lines
Quoted 3 lines
Sorry, I don't quite understand why we need to worry about this flush too much for a case which won't occur in normal condition (if everything is correct). Why we can't just consider this flush as a required action,
A translation error can be easy to reach. For instance if the guest does not program the Device correctly and request to access an address that is not mapped.
Yes, I understand these bits. But, I wrote that error wouldn't occur in normal condition (if everything was correct).
Quoted 11 lines
Quoted 10 lines
which needed to exit from the error state and resume stopped address translation request. The same required action as "clearing error status flags" before. We are not trying to understand, why is it so necessary to clear error flags when error happens, or can we end up without clearing it, for example. We just follow what described in document. The same, I think, we have for that flush, if described, then should be followed. Looks like this flush acts as a trigger to unblock stopped transaction in that particular case.
Quoted 5 lines
What will actually happen if the transaction fail again? For instance, if the IOVA was not mapped. Will you receive the interrupt again? If so, are you going to make the flush again and again until the guest is killed?
This is a good question. I think, if address is not mapped, the transaction will fail again and we will get the interrupt again. Not sure, until the guest is killed or until the driver in the guest detects timeout and cancels DMA. Let's consider the worst case, until the guest is killed.
So my questions are what do you think would be the proper driver's behavior in that case? Do nothing and don't even try to resolve error condition/unblock translation at the first page fault, or give it a few attempts, or unblock every time. How does the SMMU driver act in such situation?
Quite clear, if we get a fault, then address is not mapped. I think, it can be both: by issuing wrong address (baggy driver, malicious driver) or by race (unlikely). If this is the real race (device hits brake-before-make, for example), we could give it another attempt, for example. Looks like we need some mechanism to deploy faulted address to P2M code (which manages page table) to analyze? Or it is not worth doing that?
Quoted 5 lines
Quoted 5 lines
Different H/W could have different restoring sequences. Some H/W requires just clearing error status, other H/W requires full re-initialization in a specific order to recover from the error state.
Quoted 7 lines
Quoted 1 line
Please correct me if I am wrong.
I am not confident to accept any code that I don't understand or I don't find sensible. As I pointed out in my previous e-mail, this hasn't reached upstream so something looks quite fishy here.
As I answered in previous e-mail, I hope, we will be able to clarify a reason why this hasn't reached upstream.
Quoted 1 line
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
(+ Robin)
Quoted 1 line · Oleksandr Tyshchenko
Hi Julien.
Hi,
Quoted 115 lines
Sorry for the possible format issues.
чт, 8 авг. 2019 г., 23:32 Julien Grall <[email protected] <mailto:[email protected]>>:
Hi Oleksandr,
On 8/8/19 8:29 PM, Oleksandr wrote:
>>>>>>
>>>>>>> Sorry for the possible format issues.
>>>>>>>
>>>>>>>
>>>>>>> > No, it is not disabled. But,ipmmu_irq()uses another >>>>>>>mmu->lock. So, I
>>>>>>> > think, there won't be a deadlock.
>>>>>>> >
>>>>>>> > Or I really missed something?
>>>>>>> >
>>>>>>> > If we worry aboutipmmu_tlb_invalidate()which is called >>>>>>> here (to
>>>>>>> > perform a flush by request from P2M code, which manages a >>>>>>> page table)
>>>>>>> > and from the irq handler (to perform a flush to resume >>>>>>> address
>>>>>>> > translation), I could use a tasklet to schedule >>>>>>>ipmmu_tlb_invalidate()
>>>>>>> > from the irq handler then. This way we would get this >>>>>>> serialized. What
>>>>>>> > do you think?
>>>>>>>
>>>>>>> I am afraid a tasklet is not an option. You need to perform >>>>>>> the TLB
>>>>>>> flush when requested otherwise you are introducing a security >>>>>>> issue.
>>>>>>>
>>>>>>> This is because as soon as a region is unmapped in the page >>>>>>> table, we
>>>>>>> remove the drop the reference on any page backing that >>>>>>> region. When the
>>>>>>> reference is dropped to zero, the page can be reallocated to >>>>>>> another
>>>>>>> domain or even Xen. If the TLB flush happen after, then the >>>>>>> guest may
>>>>>>> still be able to access the page for a short time if the >>>>>>> translation has
>>>>>>> been cached by the any TLB (IOMMU, Processor). >>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I understand this. I am not proposing to delay a requested by P2M >>>>>>> code TLB flush in any case. I just propose to issue TLB flush >>>>>>> (which we have to perform in case of page faults, to resolve >>>>>>> error condition and resume translations) from a tasklet rather >>>>>>> than from interrupt handler directly. This is the TLB flush I am >>>>>>> speaking about:
>>>>>>>
>>>>>>>
https://github.com/otyshchenko1/xen/blob/ipmmu_upstream2/xen/drivers/passthrough/arm/ipmmu-vmsa.c#L598>>>>>>>
>>>>>>>
>>>>>>> Sorry if I was unclear.
>>>>>>
>>>>>> My mistake, I misread what you wrote.
>>>>>>
>>>>>> I found the flush in the renesas-bsp and not Linux upstream but it >>>>>> is not clear why this is actually required. You are not fixing any >>>>>> translation error. So what this flush will do?
>>>>>>
>>>>>> Regarding the placement of the flush, then if you execute in a >>>>>> tasklet it will likely be done later on when the IRQ has been >>>>>> acknowledge. What's the implication to delay it?
>>>>>
>>>>>
>>>>> Looks like, there is no need to put this flush into a tasklet. As I >>>>> understand from Shimoda-san's answer it is OK to call flush here. >>>>>
>>>>> So, my worry about callingipmmu_tlb_invalidate()directly from the >>>>> interrupt handler is not actual anymore.
>>>>> ----------
>>>>> This is my understanding regarding the flush purpose here. This >>>>> code, just follows the TRM, no more no less,
>>>>> which mentions about a need to flush TLB after clearing error >>>>> status register and updating a page table entries (which, I assume, >>>>> means to resolve a reason why translation/page fault error actually >>>>> have happened) to resume address translation request. >>>>
>>>> Well, I don't have the TRM... so my point of reference is Linux. Why >>>> does upstream not do the TLB flush?
>>>
>>> I have no idea regarding that. >
>>>
>>>>
>>>>
>>>> I have been told this is an errata on the IPMMU. Is it related to >>>> the series posted on linux-iommu [1]?
>>>
>>> I don't think, the TLB flush we are speaking about, is related to >>> that series [1] somehow. This TLB flush, I think, is just the last >>> step in a sequence of actions which should be performed when the >>> error occurs, no more no less. This is how I understand this. >>
>> If you have to flush the TLBs in the IRQ context then something has >> gone really wrong.
>>
>> I don't deny that Break-Before-Make is an issue. However, if it is >> handled correctly in the P2M code. You should only be there because >> there are no mapping in the TLBs for the address accessed. So flushing >> the TLBs should be unnecessary, unless your TLB is also caching >> invalid entry?
>
> Sorry, I don't quite understand why we need to worry about this flush > too much for a case which won't occur in normal condition (if everything > is correct). Why we can't just consider this flush as a required action,
Quoted 6 lines
A translation error can be easy to reach. For instance if the guest does not program the Device correctly and request to access an address that is not mapped.
Yes, I understand these bits. But, I wrote that error wouldn't occur in normal condition (if everything was correct).
I don't understand your point here. Whether this is in an error path or correct path, we should be able to understand the reason behind it. Otherwise, error path would become the wild west...
Quoted 20 lines
> which needed to exit from the error state and resume stopped address > translation request. The same required action as "clearing error status > flags" before. We are not trying to understand, why is it so necessary > to clear error flags when error happens, or can we end up without > clearing it, for example. We just follow what described in document. The > same, I think, we have for that flush, if described, then should be > followed. Looks like this flush acts as a trigger to unblock stopped > transaction in that particular case.
What will actually happen if the transaction fail again? For instance, if the IOVA was not mapped. Will you receive the interrupt again? If so, are you going to make the flush again and again until the guest is killed?
Quoted 3 lines
This is a good question. I think, if address is not mapped, the transaction will fail again and we will get the interrupt again. Not sure, until the guest is killed or until the driver in the guest detects timeout and cancels DMA. Let's consider the worst case, until the guest is killed.
So my questions are what do you think would be the proper driver's behavior in that case? Do nothing and don't even try to resolve error condition/unblock translation at the first page fault, or give it a few attempts, or unblock every time.
I will answer back with a question here. How is the TLB flush is going to unblock anything? The more you are not fixing any error condition here... And the print "Unhandled fault" just afterwards clearly leads to think that there are very little chance the fault has been resolved.
Quoted 1 line
How does the SMMU driver act in such situation?
I have CCed Robin who knows better than me the SMMU driver. Though it is the Linux one but Xen is based on it.
From my understanding, it is implementation defined whether the SMMU supports stalling a transaction on fault. AFAICT, the current Xen driver will just terminate the transaction and therefore the client transaction behave as RAZ/WI.
Quoted 2 lines
Quite clear, if we get a fault, then address is not mapped. I think, it can be both: by issuing wrong address (baggy driver, malicious driver) or by race (unlikely). If this is the real race (device hits brake-before-make, for example), we could give it another attempt, for example. Looks like we need some mechanism to deploy faulted address to P2M code (which manages page table) to analyze? Or it is not worth doing that?
You seem to speak about break-before-make as it was an error. Break-Before-Make is just a sequence to prevent the TLB walker to cache both old and new mapping at the same time. At a given point the IOVA translation can only be:
1) The old physical address
2) No address -> result to a fault
3) The new physical address
1) and 3) should not result to a fault. 2) will result to a fault but then the TLB should not cache invalid entry, right?
In order to see 2), we always flush the TLBs after removing the old physical address.
Unfortunately, some of the IOMMUs are not able to restart transactions, Xen currently avoids to flush the TLBs after 2). So you may be able to see both mapping at the same time.
Looking at your driver, I believe you would have the flag IMSTR.MHIT (multiple tlb hits) set because this is the condition we are trying to prevent with break-before-make. The comment in the code leads to think this is a fault error, so I am not sure why you would recover here...
If your IOMMU is able to stall transaction, then it would be best if we properly handle break-before-make with it.
Overall, it feels to me the TLB flush is here for a different reason.
Quoted 9 lines
>
> Different H/W could have different restoring sequences. Some H/W > requires just clearing error status, other H/W requires full > re-initialization in a specific order to recover from the error state. >
> Please correct me if I am wrong.
Quoted 6 lines
I am not confident to accept any code that I don't understand or I don't find sensible. As I pointed out in my previous e-mail, this hasn't reached upstream so something looks quite fishy here.
As I answered in previous e-mail, I hope, we will be able to clarify a reason why this hasn't reached upstream.
Thank you.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Hi, Julien
Quoted 7 lines
Quoted 7 lines
What will actually happen if the transaction fail again? For instance, if the IOVA was not mapped. Will you receive the interrupt again? If so, are you going to make the flush again and again until the guest is killed?
Quoted 5 lines
Quoted 3 lines
This is a good question. I think, if address is not mapped, the transaction will fail again and we will get the interrupt again. Not sure, until the guest is killed or until the driver in the guest detects timeout and cancels DMA. Let's consider the worst case, until the guest is killed.
So my questions are what do you think would be the proper driver's behavior in that case? Do nothing and don't even try to resolve error condition/unblock translation at the first page fault, or give it a few attempts, or unblock every time.
I will answer back with a question here. How is the TLB flush is going to unblock anything? The more you are not fixing any error condition here... And the print "Unhandled fault" just afterwards clearly leads to think that there are very little chance the fault has been resolved.
Now I understand your point. This really makes sense.
Quoted 6 lines
Quoted 2 lines
How does the SMMU driver act in such situation?
I have CCed Robin who knows better than me the SMMU driver. Though it is the Linux one but Xen is based on it.
From my understanding, it is implementation defined whether the SMMU supports stalling a transaction on fault. AFAICT, the current Xen driver will just terminate the transaction and therefore the client transaction behave as RAZ/WI.
I got it. So, sounds like the client won't be able to do something bad, and we won't receive an interrupt storm here in Xen.
Quoted 11 lines
Quoted 4 lines
Quite clear, if we get a fault, then address is not mapped. I think, it can be both: by issuing wrong address (baggy driver, malicious driver) or by race (unlikely). If this is the real race (device hits brake-before-make, for example), we could give it another attempt, for example. Looks like we need some mechanism to deploy faulted address to P2M code (which manages page table) to analyze? Or it is not worth doing that?
You seem to speak about break-before-make as it was an error. Break-Before-Make is just a sequence to prevent the TLB walker to cache both old and new mapping at the same time. At a given point the IOVA translation can only be:
1) The old physical address
2) No address -> result to a fault
3) The new physical address1) and 3) should not result to a fault. 2) will result to a fault but then the TLB should not cache invalid entry, right?
right.
Quoted 8 lines
In order to see 2), we always flush the TLBs after removing the old physical address.
Unfortunately, some of the IOMMUs are not able to restart transactions, Xen currently avoids to flush the TLBs after 2). So you may be able to see both mapping at the same time.
Looking at your driver, I believe you would have the flag IMSTR.MHIT (multiple tlb hits) set because this is the condition we are trying to prevent with break-before-make. The comment in the code leads to think this is a fault error, so I am not sure why you would recover here...
If your IOMMU is able to stall transaction, then it would be best if we properly handle break-before-make with it.
Thank you for the detailed answer. I would like to say that I have never seen Multiple tlb hits error raised by IPMMU in Xen.
Quoted 2 lines
Overall, it feels to me the TLB flush is here for a different reason.
I will drop this TLB flush from interrupt handler until clarified.
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
In reply to Julien GrallHi, Julien.
Quoted 11 lines
Quoted 5 lines
I am afraid, I don't know a correct answer for this question. I would leave this to maintainers.
I just followed sample copyright notice for GPL v2 License according to the document:
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=CONTRIBUTING
The file CONTRIBUTING is only giving example of common example of license. So I think this is fine to use SPDX, the more they are already used. The only request is to put either SDPX or the full-blown text but not the two :). Lars, any objection?
I am quite in favor of SPDX because it is easier to find out the license. With the full-blown text, the text may slightly vary between licenses. For instance, the only difference between GPLv2 and GPLv2+ is ",or (at your option) any later version". I let you imagine how it can be easy to miss it when reviewing ;).
We had a discussion last year about using SPDX in Xen code base but I never got the time to formally suggest it.
I tried to locate files in Xen where SPDX is used. After finding only nospec.h I got an incorrect opinion this is not popular in Xen))
Just to clarify. So the title for the driver should be the following (if there are no objections):
// SPDX-License-Identifier: GPL-2.0
/*
* xen/drivers/passthrough/arm/ipmmu-vmsa.c
*
* Driver for the Renesas IPMMU-VMSA found in R-Car Gen3 SoCs.
*
* Copyright (C) 2014-2019 Renesas Electronics Corporation
*
* The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU)
* which provides address translation and access protection functionalities
* to processing units and interconnect networks.
*
* Please note, current driver is supposed to work only with newest R-Car Gen3
* SoCs revisions which IPMMU hardware supports stage 2 translation table format
* and is able to use CPU's P2M table as is.
*
* Based on Linux's IPMMU-VMSA driver from Renesas BSP:
* drivers/iommu/ipmmu-vmsa.c
* you can found at:
* url: git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git
* branch: v4.14.75-ltsi/rcar-3.9.6
* commit: e206eb5b81a60e64c35fbc3a999b1a0db2b98044
* and Xen's SMMU driver:
* xen/drivers/passthrough/arm/smmu.c
*
* Copyright (C) 2016-2019 EPAM Systems Inc.
*/
Answer to myself:
Looks like, the same I have to do with all newly added files in this series (iommu_fwspec, etc).
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 12 lines
+ /*
+ * Destroy Root IPMMU domain which context is mapped to this Xen domain
+ * if exits.
+ */
+ if (xen_domain->root_domain)
+ipmmu_free_root_domain(xen_domain->root_domain);xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +5
-0+ + spin_unlock(&xen_domain->lock); + + /* +* We assume that all master devices have already been Quoted 6 lines
detached from
+ * this Xen domain and there must be no associated Cache IPMMU domains
+ * in use.
+ */
+ ASSERT(list_empty(&xen_domain->cache_domains));I think this should be in the spin lock held by
&xen_domain->lock.
Quoted 5 lines
Quoted 1 line
OK. Will put
spin_unlockafter it.The
spin_lockis actually pointless here. This is done when the domain is destroyed, so nobody should touch it.If you think concurrent access can still happen, then you are going to be in deep trouble as you free the
xen_domain(and therefore the spinlock) below :).
Indeed, this is pointless. We don't really expect any other operations with the domain which is being destroyed. No assign/deassign devices, no flush, no map, nothing...
Quoted 2 lines
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
In reply to Julien GrallOn 07/08/2019, 20:15, "Julien Grall" <[email protected]> wrote:
(+ Lars)
Hi,
On 8/7/19 5:01 PM, Oleksandr wrote:
>>> + * you can found at:
>>> + * url:
>>> git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git
>>> + * branch: v4.14.75-ltsi/rcar-3.9.6
>>> + * commit: e206eb5b81a60e64c35fbc3a999b1a0db2b98044
>>> + * and Xen's SMMU driver:
>>> + * xen/drivers/passthrough/arm/smmu.c
>>> + *
>>> + * Copyright (C) 2016-2019 EPAM Systems Inc.
>>> + *
>>> + * This program is free software; you can redistribute it and/or >>> + * modify it under the terms and conditions of the GNU General Public >>> + * License, version 2, as published by the Free Software Foundation. >>> + *
>>> + * This program is distributed in the hope that it will be useful, >>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of >>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU >>> + * General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public >>> + * License along with this program; If not, see
>>> <http://www.gnu.org/licenses/>.
>> I don't know that Xen license description rule, but since a few source >> files have
>> SPDX-License-Identifier, can we also use it on the driver? >
> I am afraid, I don't know a correct answer for this question. I would > leave this to maintainers.
>
> I just followed sample copyright notice for GPL v2 License according to > the document:
>
> http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=CONTRIBUTING
The file CONTRIBUTING is only giving example of common example of license. So I think this is fine to use SPDX, the more they are already used. The only request is to put either SDPX or the full-blown text but not the two :). Lars, any objection?
I am quite in favor of SPDX because it is easier to find out the license. With the full-blown text, the text may slightly vary between licenses. For instance, the only difference between GPLv2 and GPLv2+ is ",or (at your option) any later version". I let you imagine how it can be easy to miss it when reviewing ;).
We had a discussion last year about using SPDX in Xen code base but I never got the time to formally suggest it.
I did not push it either.
In the past one of the committers had major objections against SPDX, but after a conversation last year and changes to the latest version of SPDX he dropped these.
The only remaining objection was to have both SPDX identifier AND a license in the same file. The argument against it is: what does it mean if they contradict each other? To be fair that is a valid concern.
I am not sure it is a good idea to introduce SPDX piecemeal. It would be much better to
a) agree it
b) transform the codebase using a tool
rather than introducing it piecemeal
Lars
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
In reply to OleksandrHi Oleksandr-san,
Quoted 6 lines
From: Oleksandr, Sent: Thursday, August 8, 2019 1:01 AM
Hi, Shimoda-san.
Thank you for the review.
You're welcome.
<snip>
Quoted 14 lines
Quoted 12 lines
+/* Xen IOMMU ops */
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 811
Quoted 8 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8
-0+static int __must_check ipmmu_iotlb_flush_all (struct domain *d) +{ +struct ipmmu_vmsa_xen_domain *xen_domain =dom_iommu (d)->arch.priv; + +if ( !xen_domain || !xen_domain->root_domain ) +return 0 ; + +spin_lock (&xen_domain->lock);Is local irq is already disabled here?
If no, you should usespin_lock_irqsave()because theipmmu_irq()also gets the lock.
Quoted 4 lines
No, it is not disabled. But,
ipmmu_irq()uses anothermmu->lock. So, I think, there won't be a deadlock.Or I really missed something?
You're correct. I didn't realized that ipmmu_irq() used another mmu->lock.
Quoted 6 lines
If we worry about
ipmmu_tlb_invalidate()which is called here (to perform a flush by request from P2M code, which manages a page table) and from the irq handler (to perform a flush to resume address translation), I could use a tasklet to scheduleipmmu_tlb_invalidate()from the irq handler then. This way we would get this serialized. What do you think?
I just concerned about a dead-lock issue by recursive spin locks. So, calling ipmmu_tlb_invalidate() here is OK, I think.
Best regards,
Yoshihiro Shimoda
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
In reply to Oleksandr TyshchenkoHi Oleksandr,
Quoted 8 lines · Oleksandr Tyshchenko
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +8
-0+static int ipmmu_iommu_domain_init (struct domain *d) +{ +struct ipmmu_vmsa_xen_domain *xen_domain; + + xen_domain = xzalloc(struct ipmmu_vmsa_xen_domain); +if ( !xen_domain ) +return -ENOMEM ; +
xen/drivers/passthrough/arm/ipmmu-vmsa.c line 1129
Quoted 10 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +10
-0+spin_lock_init (&xen_domain->lock); +INIT_LIST_HEAD (&xen_domain->cache_domains); +/* +* We don't create Root IPMMU domain here, it will be created on demand +* only, when attaching the first master device to this Xen domain in +* ipmmu_assign_device(). +* xen_domain->root_domain = NULL; +*/ + +dom_iommu (d)->arch.priv = xen_domain;
While looking at other part of Xen I realized you don't set IOMMU_FEAT_COHERENT_WALK. Does it mean the IOMMU walker does not support coherent walk (i.e snooping the cache)?
Note that when this feature is not set, the p2m code will require to clean each P2M entry when updated. So if the IPMMU supports coherent walk, I would strongly suggest to set the flag :).
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH V2 6/6] iommu/arm: Add Renesas IPMMU-VMSA support
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi Julien.
Quoted 16 lines · Oleksandr Tyshchenko
Quoted 14 lines
xen/drivers/passthrough/arm/ipmmu-vmsa.c new file +12
-0+static int ipmmu_iommu_domain_init (struct domain *d) +{ + struct ipmmu_vmsa_xen_domain *xen_domain; + + xen_domain = xzalloc(struct ipmmu_vmsa_xen_domain); + if ( !xen_domain ) + return -ENOMEM; + + spin_lock_init(&xen_domain->lock); + INIT_LIST_HEAD(&xen_domain->cache_domains); + /* +* We don't create Root IPMMU domain here, it will be created on demand
+ * only, when attaching the first master device to this Xen
Quoted 8 lines
Quoted 6 lines
domain in
+ *ipmmu_assign_device().
+ *xen_domain->root_domain= NULL;
+ */
+
+ dom_iommu(d)->arch.priv =xen_domain;While looking at other part of Xen I realized you don't set
IOMMU_FEAT_COHERENT_WALK. Does it mean the IOMMU walker does not support coherent walk (i.e snooping the cache)?
*AFAIK*, not supported.
Linux driver reports coherent_walk is not supported as well.
Quoted 3 lines
Note that when this feature is not set, the p2m code will require to clean each P2M entry when updated. So if the IPMMU supports coherent walk, I would strongly suggest to set the flag :).
When playing with non-shared IOMMU in Xen (two years ago), I noticed that I had forgotten to use clean_dcache after updating a page table entry. I could face faults when shattering superpages for example. Once I added it, the faults went away completely.
So, leave IOMMU_FEAT_COHERENT_WALK in disabled state, but will keep your suggestion in mind.
Quoted 3 lines
Cheers,
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Hello, all.
Forgot to mention that an additional patch from Xen staging is needed, otherwise Xen may crash at the early stage:ead6b9f78355e8d366e0c80c4a73fa7fbd6d26cc
"xen/arm: cpuerrata: Align a virtual address before unmap"
Signature · 9 lines
Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 2 lines · Oleksandr
Hello, all.
Hi,
Quoted 4 lines
Forgot to mention that an additional patch from Xen staging is needed, otherwise Xen may crash at the early stage:
ead6b9f78355e8d366e0c80c4a73fa7fbd6d26cc
"xen/arm: cpuerrata: Align a virtual address before unmap"
This patch is already merged in Xen :).
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel