IOMMU probe deferral support

This series calls the dma ops configuration for the devices at a generic place so that it works for all busses. The dma_configure_ops for a device is now called during the device_attach callback just before the probe of the bus/driver is called. Similarly dma_deconfigure is called during device/driver_detach path.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

Rebased the series against mainline 4.11-rc1. Applies and builds cleanly against mainline and linux-next. There is a conflict with patch#9 against iommu-next, but that should go away eventually as iommu-next is rebased against 4.11-rc1.

* Tested with platform and pci devices for probe deferral and reprobe on arm64 based platform.

Previous post of this series [6].

[V9]
* Rebased on top of 4.11-rc1.

* Merged Robin's fixes for legacy binding issue, pci devices with no iommu-map property.

[V8]
* Picked up all the acks and tested tags from Marek and Hanjun for DT and ACPI patches respectively, since no functional changes was done.

* Addressed Minor comments Sinan and Bjorn.

* Added Robin's fix for fixing the deferencing NULL for of_iommu_table after init in patch #2.

* Rebased it on top of linux-next

[V7]
* Updated the subject and commit log for patch #6 as per comments from Lorenzo. No functional changes.

[V6]
* Fixed a bug in dma_configure function pointed out by Robin.
* Reordered the patches as per comments from Robin and Lorenzo.
* Added Tags.

[V5]
* Reworked the pci configuration code hanging outside and pushed it to dma_configure as in PATCH#5,6,7. Also added a couple of patches that Lorenzo provided for correcting the Probe deferring mechanism in case of ACPI devices from here [5].

[V4]
* Took the reworked patches [2] from Robin's branch and rebased on top of Lorenzo's ACPI IORT ARM support series [3].

* Added the patches for moving the dma ops configuration of acpi based devices to probe time as well.
[V3]
* Removed the patch to split dma_masks/dma_ops configuration separately based on review comments that both masks and ops are required only during the device probe time.

* Reworked the series based on Generic DT bindings series.

* Added call to iommu's remove_device in the cleanup path for arm and arm64.

* Removed the notifier trick in arm64 to handle early device registration.

* Added reset of dma_ops in cleanup path for arm based on comments.

* Fixed the pci_iommu_configure path and tested with PCI device as well.

* Fixed a bug to return the correct iommu_ops from patch 7 [4] in last post.

* Fixed few other cosmetic comments.

[V2]
* Updated the Initial post to call dma_configure/deconfigure from generic code

* Added iommu add_device callback from of_iommu_configure path

[V1]
* Initial post from Laurent Pinchart [1]

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
[3] https://lkml.org/lkml/2016/11/21/141
[4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
[5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
[6] http://www.spinics.net/lists/linux-pci/msg57992.html
[7] https://www.spinics.net/lists/arm-kernel/msg556209.html

Laurent Pinchart (3):
of: dma: Move range size workaround to of_dma_get_range()
of: dma: Make of_dma_deconfigure() public
iommu: of: Handle IOMMU lookup failure with deferred probing or error

Lorenzo Pieralisi (2):
ACPI/IORT: Add function to check SMMUs drivers presence ACPI/IORT: Remove linker section for IORT entries probing

Robin Murphy (3):
iommu/of: Refactor of_iommu_configure() for error handling iommu/of: Prepare for deferred IOMMU configuration iommu/arm-smmu: Clean up early-probing workarounds

Sricharan R (3):
of/acpi: Configure dma operations at probe time for platform/amba/pci
bus devices
drivers: acpi: Handle IOMMU lookup failure with deferred probing or
error
arm64: dma-mapping: Remove the notifier trick to handle early setting
of dma_ops

 arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
 drivers/acpi/arm64/iort.c         |  40 ++++++++++-
 drivers/acpi/glue.c               |   5 --
 drivers/acpi/scan.c               |   7 +-
 drivers/base/dd.c                 |   9 +++
 drivers/base/dma-mapping.c        |  41 +++++++++++
 drivers/iommu/arm-smmu-v3.c       |  46 +-----------
 drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
 drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
 drivers/of/address.c              |  20 +++++-
 drivers/of/device.c               |  34 ++++-----
 drivers/of/platform.c             |  10 +--
 drivers/pci/probe.c               |  28 --------
 include/acpi/acpi_bus.h           |   2 +-
 include/asm-generic/vmlinux.lds.h |   1 -
 include/linux/acpi.h              |   7 +-
 include/linux/acpi_iort.h         |   3 -
 include/linux/dma-mapping.h       |   3 +
 include/linux/of_device.h         |  10 ++-
 19 files changed, 308 insertions(+), 336 deletions(-)

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

From: Robin Murphy <[email protected]>

In preparation for some upcoming cleverness, rework the control flow in of_iommu_configure() to minimise duplication and improve the propogation of errors. It's also as good a time as any to switch over from the now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic IOMMU instance interface directly.

---
 drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 30 deletions(-)

index 2683e9f..8f4e599 100644
@@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static const struct iommu_ops
+*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
+{
+	const struct iommu_ops *ops;
+	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
+	int err;
+
+	ops = iommu_ops_from_fwnode(fwnode);
+	if (!ops || !ops->of_xlate)
+		return NULL;
+
+	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (err)
+		return ERR_PTR(err);
+
+	err = ops->of_xlate(dev, iommu_spec);
+	if (err)
+		return ERR_PTR(err);
+
+	return ops;
+}
+
 static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	struct of_phandle_args *iommu_spec = data;

@@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 }
 
 static const struct iommu_ops
-*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
+*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
 {
 	const struct iommu_ops *ops;
 	struct of_phandle_args iommu_spec;
+	int err;
 
 	/*
 	 * Start by tracing the RID alias down the PCI topology as

@@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 	 * bus into the system beyond, and which IOMMU it ends up at.
 	 */
 	iommu_spec.np = NULL;
-	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
-			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
-		return NULL;
+	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
+			     "iommu-map-mask", &iommu_spec.np,
+			     iommu_spec.args);
+	if (err)
+		return err == -ENODEV ? NULL : ERR_PTR(err);
 
-	ops = iommu_ops_from_fwnode(&iommu_spec.np->fwnode);
-	if (!ops || !ops->of_xlate ||
-	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
-	    ops->of_xlate(&pdev->dev, &iommu_spec))
-		ops = NULL;
+	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
 
 	of_node_put(iommu_spec.np);
 	return ops;
 }
 
-const struct iommu_ops *of_iommu_configure(struct device *dev,
-					   struct device_node *master_np)
+static const struct iommu_ops
+*of_platform_iommu_init(struct device *dev, struct device_node *np)
 {
 	struct of_phandle_args iommu_spec;
-	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
-
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
 	 * See the `Notes:' section of
 	 * Documentation/devicetree/bindings/iommu/iommu.txt
 	 */
-	while (!of_parse_phandle_with_args(master_np, "iommus",
-					   "#iommu-cells", idx,
-					   &iommu_spec)) {
-		np = iommu_spec.np;
-		ops = iommu_ops_from_fwnode(&np->fwnode);
-
-		if (!ops || !ops->of_xlate ||
-		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
-		    ops->of_xlate(dev, &iommu_spec))
-			goto err_put_node;
-
-		of_node_put(np);
+	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+					   idx, &iommu_spec)) {
+		ops = of_iommu_xlate(dev, &iommu_spec);
+		of_node_put(iommu_spec.np);
 		idx++;
+		if (IS_ERR_OR_NULL(ops))
+			break;
 	}
 
 	return ops;
+}
+
+const struct iommu_ops *of_iommu_configure(struct device *dev,
+					   struct device_node *master_np)
+{
+	const struct iommu_ops *ops;
+
+	if (!master_np)
+		return NULL;
+
+	if (dev_is_pci(dev))
+		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
+	else
+		ops = of_platform_iommu_init(dev, master_np);
 
-err_put_node:
-	of_node_put(np);
-	return NULL;
+	return IS_ERR(ops) ? NULL : ops;
 }
 
 static int __init of_iommu_init(void)

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/[email protected]>

Failures to look up an IOMMU when parsing the DT iommus property need to be handled separately from the .of_xlate() failures to support deferred probing.

The lack of a registered IOMMU can be caused by the lack of a driver for the IOMMU, the IOMMU device probe not having been performed yet, having been deferred, or having failed.

The first case occurs when the device tree describes the bus master and IOMMU topology correctly but no device driver exists for the IOMMU yet or the device driver has not been compiled in. Return NULL, the caller will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus master device as well. A mechanism to either configure the bus master device without an IOMMU or to fail the bus master device probe depending on whether the IOMMU is optional or mandatory would be a good enhancement.

---
 drivers/base/dma-mapping.c | 5 +++--
 drivers/iommu/of_iommu.c   | 4 ++--
 drivers/of/device.c        | 7 ++++++-
 include/linux/of_device.h  | 9 ++++++---
 4 files changed, 17 insertions(+), 8 deletions(-)

index 449b948..82bd45c 100644
@@ -353,6 +353,7 @@ int dma_configure(struct device *dev)
 {
 	struct device *bridge = NULL, *dma_dev = dev;
 	enum dev_dma_attr attr;
+	int ret = 0;
 
 	if (dev_is_pci(dev)) {
 		bridge = pci_get_host_bridge_device(to_pci_dev(dev));

@@ -363,7 +364,7 @@ int dma_configure(struct device *dev)
 	}
 
 	if (dma_dev->of_node) {
-		of_dma_configure(dev, dma_dev->of_node);
+		ret = of_dma_configure(dev, dma_dev->of_node);
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)

@@ -373,7 +374,7 @@ int dma_configure(struct device *dev)
 	if (bridge)
 		pci_put_host_bridge_device(bridge);
 
-	return 0;
+	return ret;
 }
 
 void dma_deconfigure(struct device *dev)

index c8be889..9f44ee8 100644
@@ -236,7 +236,7 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 			ops = ERR_PTR(err);
 	}
 
-	return IS_ERR(ops) ? NULL : ops;
+	return ops;
 }
 
 static int __init of_iommu_init(void)

@@ -247,7 +247,7 @@ static int __init of_iommu_init(void)
 	for_each_matching_node_and_match(np, matches, &match) {
 		const of_iommu_init_fn init_fn = match->data;
 
-		if (init_fn(np))
+		if (init_fn && init_fn(np))
 			pr_err("Failed to initialise IOMMU %s\n",
 				of_node_full_name(np));
 	}

index c17c19d..ba51ca6 100644
@@ -82,7 +82,7 @@ int of_device_add(struct platform_device *ofdev)
  * can use a platform bus notifier and handle BUS_NOTIFY_ADD_DEVICE events
  * to fix up DMA configuration.
  */
-void of_dma_configure(struct device *dev, struct device_node *np)
+int of_dma_configure(struct device *dev, struct device_node *np)
 {
 	u64 dma_addr, paddr, size;
 	int ret;

@@ -129,10 +129,15 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		coherent ? " " : " not ");
 
 	iommu = of_iommu_configure(dev, np);
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
+
 	dev_dbg(dev, "device is%sbehind an iommu\n",
 		iommu ? " " : " not ");
 
 	arch_setup_dma_ops(dev, dma_addr, size, iommu, coherent);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 

index af98455..2cacdd8 100644
@@ -55,7 +55,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 	return of_node_get(cpu_dev->of_node);
 }
 
-void of_dma_configure(struct device *dev, struct device_node *np);
+int of_dma_configure(struct device *dev, struct device_node *np);
 void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 

@@ -104,8 +104,11 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 {
 	return NULL;
 }
-static inline void of_dma_configure(struct device *dev, struct device_node *np)
-{}
+
+static inline int of_dma_configure(struct device *dev, struct device_node *np)
+{
+	return 0;
+}
 static inline void of_dma_deconfigure(struct device *dev)
 {}
 #endif /* CONFIG_OF */

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Maybe it is the same issue reported for VFIO, but virtio-pci is broken with v8 of this series. Bisecting blames this commit which looks like it hasn't changeed.

Rob

P.S. Doesn't look like you have copied the DT maintainers nor list for the DT changes.

v8 managed to break *all* PCI devices which didn't have an associated "iommu-map". The problem was actually in patch 2 (and is fixed in this version), but it only hit at this point once we start propagating the erroneous error code back to the driver probe.

Robin.

Hi,

Ya, as Robin mentioned it was broken at patch #2 and comes out after this patch. I am going to repost this series with couple of more fixes added as well.

Ha, really sorry about that. will add it.

Regards,
Sricharan

"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a 
member of Code Aurora Forum, hosted by The Linux Foundation

From: Robin Murphy <[email protected]>

IOMMU configuration represents unchanging properties of the hardware, and as such should only need happen once in a device's lifetime, but the necessary interaction with the IOMMU device and driver complicates exactly when that point should be.

Since the only reasonable tool available for handling the inter-device dependency is probe deferral, we need to prepare of_iommu_configure() to run later than it is currently called (i.e. at driver probe rather than device creation), to handle being retried, and to tell whether a not-yet present IOMMU should be waited for or skipped (by virtue of having declared a built-in driver or not).

---
 drivers/iommu/of_iommu.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

index 8f4e599..c8be889 100644
@@ -96,6 +96,19 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static bool of_iommu_driver_present(struct device_node *np)
+{
+	/*
+	 * If the IOMMU still isn't ready by the time we reach init, assume
+	 * it never will be. We don't want to defer indefinitely, nor attempt
+	 * to dereference __iommu_of_table after it's been freed.
+	 */
+	if (system_state > SYSTEM_BOOTING)
+		return false;
+
+	return of_match_node(&__iommu_of_table, np);
+}
+
 static const struct iommu_ops
 *of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
 {

@@ -104,12 +117,20 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 	int err;
 
 	ops = iommu_ops_from_fwnode(fwnode);
-	if (!ops || !ops->of_xlate)
+	if ((ops && !ops->of_xlate) ||
+	    (!ops && !of_iommu_driver_present(iommu_spec->np)))
 		return NULL;
 
 	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
 	if (err)
 		return ERR_PTR(err);
+	/*
+	 * The otherwise-empty fwspec handily serves to indicate the specific
+	 * IOMMU device we're waiting for, which will be useful if we ever get
+	 * a proper probe-ordering dependency mechanism in future.
+	 */
+	if (!ops)
+		return ERR_PTR(-EPROBE_DEFER);
 
 	err = ops->of_xlate(dev, iommu_spec);
 	if (err)

@@ -186,14 +207,34 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 					   struct device_node *master_np)
 {
 	const struct iommu_ops *ops;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 
 	if (!master_np)
 		return NULL;
 
+	if (fwspec) {
+		if (fwspec->ops)
+			return fwspec->ops;
+
+		/* In the deferred case, start again from scratch */
+		iommu_fwspec_free(dev);
+	}
+
 	if (dev_is_pci(dev))
 		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
 	else
 		ops = of_platform_iommu_init(dev, master_np);
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
 
 	return IS_ERR(ops) ? NULL : ops;
 }

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

From: Laurent Pinchart <[email protected]>

Invalid dma-ranges values should be worked around when retrieving the DMA range in of_dma_get_range(), not by all callers of the function. This isn't much of a problem now that we have a single caller, but that situation will change when moving DMA configuration to device probe time.

---
 drivers/of/address.c | 20 ++++++++++++++++++--
 drivers/of/device.c  | 15 ---------------
 2 files changed, 18 insertions(+), 17 deletions(-)

index 02b2903..6aeb816 100644
@@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  *	CPU addr (phys_addr_t)	: pna cells
  *	size			: nsize cells
  *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
  */
 int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
 {

@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
 	*dma_addr = dmaaddr;
 
 	*size = of_read_number(ranges + naddr + pna, nsize);
+	/*
+	 * DT nodes sometimes incorrectly set the size as a mask. Work around
+	 * those incorrect DT by computing the size as mask + 1.
+	 */
+	if (*size & 1) {
+		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+			__func__, *size, np->full_name);
+		*size = *size + 1;
+	}
+
+	if (!*size) {
+		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+		       __func__, np->full_name);
+		ret = -EINVAL;
+		goto out;
+	}
 
 	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
 		 *dma_addr, *paddr, *size);

index b1e6beb..09dedd0 100644
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		size = dev->coherent_dma_mask + 1;
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
-
-		/*
-		 * Add a work around to treat the size as mask + 1 in case
-		 * it is defined in DT as a mask.
-		 */
-		if (size & 1) {
-			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
-				 size);
-			size = size + 1;
-		}
-
-		if (!size) {
-			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
-			return;
-		}
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

From: Laurent Pinchart <[email protected]>

As part of moving DMA initializing to probe time the of_dma_deconfigure() function will need to be called from different source files. Make it public and move it to drivers/of/device.c where the of_dma_configure() function is.

---
 drivers/of/device.c       | 12 ++++++++++++
 drivers/of/platform.c     |  5 -----
 include/linux/of_device.h |  3 +++
 3 files changed, 15 insertions(+), 5 deletions(-)

index 09dedd0..c17c19d 100644
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev:	Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+	arch_teardown_dma_ops(dev);
+}
+
 int of_device_register(struct platform_device *pdev)
 {
 	device_initialize(&pdev->dev);

index 5dfcc96..5344db5 100644
@@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
 }
 EXPORT_SYMBOL(of_device_alloc);
 
-static void of_dma_deconfigure(struct device *dev)
-{
-	arch_teardown_dma_ops(dev);
-}
-
 /**
  * of_platform_device_create_pdata - Alloc, initialize and register an of_device
  * @np: pointer to node to create device for

index c12dace..af98455 100644
@@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 
 void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
 static inline int of_driver_match_device(struct device *dev,

@@ -105,6 +106,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
 {}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
 #endif /* CONFIG_OF */
 
 #endif /* _LINUX_OF_DEVICE_H */

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

From: Lorenzo Pieralisi <[email protected]>

The IOMMU probe deferral implementation requires a mechanism to detect if drivers for SMMU components are built-in in the kernel to detect whether IOMMU configuration for a given device should be deferred (ie SMMU drivers present but still not probed) or not (drivers not present).

Add a simple function to IORT to detect if SMMU drivers for SMMU components managed by IORT are built-in in the kernel.

---
 drivers/acpi/arm64/iort.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

index 4a5bb96..3dd9ec3 100644
@@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
 	return ret;
 }
 
+static inline bool iort_iommu_driver_enabled(u8 type)
+{
+	switch (type) {
+	case ACPI_IORT_NODE_SMMU_V3:
+		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
+	case ACPI_IORT_NODE_SMMU:
+		return IS_BUILTIN(CONFIG_ARM_SMMU);
+	default:
+		pr_warn("IORT node type %u does not describe an SMMU\n", type);
+		return false;
+	}
+}
+
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 					struct acpi_iort_node *node,
 					u32 streamid)

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Configuring DMA ops at probe time will allow deferring device probe when the IOMMU isn't available yet. The dma_configure for the device is now called from the generic device_attach callback just before the bus/driver probe is called. This way, configuring the DMA ops for the device would be called at the same place for all bus_types, hence the deferred probing mechanism should work for all buses as well.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

This patch changes the dma ops configuration to probe time for both OF and ACPI based platform/amba/pci bus devices.

---
 drivers/acpi/glue.c         |  5 -----
 drivers/base/dd.c           |  9 +++++++++
 drivers/base/dma-mapping.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 drivers/of/platform.c       |  5 +----
 drivers/pci/probe.c         | 28 ----------------------------
 include/linux/dma-mapping.h |  3 +++
 6 files changed, 53 insertions(+), 37 deletions(-)

index fb19e1c..c05f241 100644
@@ -176,7 +176,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	struct list_head *physnode_list;
 	unsigned int node_id;
 	int retval = -EINVAL;
-	enum dev_dma_attr attr;
 
 	if (has_acpi_companion(dev)) {
 		if (acpi_dev) {

@@ -233,10 +232,6 @@ int acpi_bind_one(struct device *dev, struct acpi_device *acpi_dev)
 	if (!has_acpi_companion(dev))
 		ACPI_COMPANION_SET(dev, acpi_dev);
 
-	attr = acpi_get_dma_attr(acpi_dev);
-	if (attr != DEV_DMA_NOT_SUPPORTED)
-		acpi_dma_configure(dev, attr);
-
 	acpi_physnode_link_name(physical_node_name, node_id);
 	retval = sysfs_create_link(&acpi_dev->dev.kobj, &dev->kobj,
 				   physical_node_name);

index a1fbf55..4882f06 100644
@@ -19,6 +19,7 @@
 
 #include <linux/device.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/module.h>
 #include <linux/kthread.h>
 #include <linux/wait.h>

@@ -356,6 +357,10 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	if (ret)
 		goto pinctrl_bind_failed;
 
+	ret = dma_configure(dev);
+	if (ret)
+		goto dma_failed;
+
 	if (driver_sysfs_add(dev)) {
 		printk(KERN_ERR "%s: driver_sysfs_add(%s) failed\n",
 			__func__, dev_name(dev));

@@ -417,6 +422,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 	goto done;
 
 probe_failed:
+	dma_deconfigure(dev);
+dma_failed:
 	if (dev->bus)
 		blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
 					     BUS_NOTIFY_DRIVER_NOT_BOUND, dev);

@@ -826,6 +833,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
 			drv->remove(dev);
 
 		device_links_driver_cleanup(dev);
+		dma_deconfigure(dev);
+
 		devres_release_all(dev);
 		dev->driver = NULL;
 		dev_set_drvdata(dev, NULL);

index efd71cf..449b948 100644
@@ -7,9 +7,11 @@
  * This file is released under the GPLv2.
  */
 
+#include <linux/acpi.h>
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 

@@ -341,3 +343,41 @@ void dma_common_free_remap(void *cpu_addr, size_t size, unsigned long vm_flags)
 	vunmap(cpu_addr);
 }
 #endif
+
+/*
+ * Common configuration to enable DMA API use for a device
+ */
+#include <linux/pci.h>
+
+int dma_configure(struct device *dev)
+{
+	struct device *bridge = NULL, *dma_dev = dev;
+	enum dev_dma_attr attr;
+
+	if (dev_is_pci(dev)) {
+		bridge = pci_get_host_bridge_device(to_pci_dev(dev));
+		dma_dev = bridge;
+		if (IS_ENABLED(CONFIG_OF) && dma_dev->parent &&
+		    dma_dev->parent->of_node)
+			dma_dev = dma_dev->parent;
+	}
+
+	if (dma_dev->of_node) {
+		of_dma_configure(dev, dma_dev->of_node);
+	} else if (has_acpi_companion(dma_dev)) {
+		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
+		if (attr != DEV_DMA_NOT_SUPPORTED)
+			acpi_dma_configure(dev, attr);
+	}
+
+	if (bridge)
+		pci_put_host_bridge_device(bridge);
+
+	return 0;
+}
+
+void dma_deconfigure(struct device *dev)
+{
+	of_dma_deconfigure(dev);
+	acpi_dma_deconfigure(dev);
+}

index 5344db5..2aa4ebb 100644
@@ -22,6 +22,7 @@
 #include <linux/slab.h>
 #include <linux/of_address.h>
 #include <linux/of_device.h>
+#include <linux/of_iommu.h>
 #include <linux/of_irq.h>
 #include <linux/of_platform.h>
 #include <linux/platform_device.h>

@@ -186,11 +187,9 @@ static struct platform_device *of_platform_device_create_pdata(
 
 	dev->dev.bus = &platform_bus_type;
 	dev->dev.platform_data = platform_data;
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
 	if (of_device_add(dev) != 0) {
-		of_dma_deconfigure(&dev->dev);
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}

@@ -248,7 +247,6 @@ static struct amba_device *of_amba_device_create(struct device_node *node,
 		dev_set_name(&dev->dev, "%s", bus_id);
 	else
 		of_device_make_bus_id(&dev->dev);
-	of_dma_configure(&dev->dev, dev->dev.of_node);
 
 	/* Allow the HW Peripheral ID to be overridden */
 	prop = of_get_property(node, "arm,primecell-periphid", NULL);

@@ -542,7 +540,6 @@ static int of_platform_device_destroy(struct device *dev, void *data)
 		amba_device_unregister(to_amba_device(dev));
 #endif
 
-	of_dma_deconfigure(dev);
 	of_node_clear_flag(dev->of_node, OF_POPULATED);
 	of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
 	return 0;

index dfc9a27..5a8dd43 100644
@@ -1893,33 +1893,6 @@ static void pci_set_msi_domain(struct pci_dev *dev)
 	dev_set_msi_domain(&dev->dev, d);
 }
 
-/**
- * pci_dma_configure - Setup DMA configuration
- * @dev: ptr to pci_dev struct of the PCI device
- *
- * Function to update PCI devices's DMA configuration using the same
- * info from the OF node or ACPI node of host bridge's parent (if any).
- */
-static void pci_dma_configure(struct pci_dev *dev)
-{
-	struct device *bridge = pci_get_host_bridge_device(dev);
-
-	if (IS_ENABLED(CONFIG_OF) &&
-		bridge->parent && bridge->parent->of_node) {
-			of_dma_configure(&dev->dev, bridge->parent->of_node);
-	} else if (has_acpi_companion(bridge)) {
-		struct acpi_device *adev = to_acpi_device_node(bridge->fwnode);
-		enum dev_dma_attr attr = acpi_get_dma_attr(adev);
-
-		if (attr == DEV_DMA_NOT_SUPPORTED)
-			dev_warn(&dev->dev, "DMA not supported.\n");
-		else
-			acpi_dma_configure(&dev->dev, attr);
-	}
-
-	pci_put_host_bridge_device(bridge);
-}
-
 void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 {
 	int ret;

@@ -1933,7 +1906,6 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
 	dev->dev.dma_mask = &dev->dma_mask;
 	dev->dev.dma_parms = &dev->dma_parms;
 	dev->dev.coherent_dma_mask = 0xffffffffull;
-	pci_dma_configure(dev);
 
 	pci_set_dma_max_seg_size(dev, 65536);
 	pci_set_dma_seg_boundary(dev, 0xffffffff);

index 0977317..6d08016 100644
@@ -728,6 +728,9 @@ void *dma_mark_declared_memory_occupied(struct device *dev,
 }
 #endif /* CONFIG_HAVE_GENERIC_DMA_COHERENT */
 
+int dma_configure(struct device *dev);
+void dma_deconfigure(struct device *dev);
+
 /*
  * Managed DMA API
  */

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

This is an equivalent to the DT's handling of the iommu master's probe with deferred probing when the corrsponding iommu is not probed yet. The lack of a registered IOMMU can be caused by the lack of a driver for the IOMMU, the IOMMU device probe not having been performed yet, having been deferred, or having failed.

The first case occurs when the firmware describes the bus master and IOMMU topology correctly but no device driver exists for the IOMMU yet or the device driver has not been compiled in. Return NULL, the caller will configure the device without an IOMMU.

The second and third cases are handled by deferring the probe of the bus master device which will eventually get reprobed after the IOMMU.

The last case is currently handled by deferring the probe of the bus master device as well. A mechanism to either configure the bus master device without an IOMMU or to fail the bus master device probe depending on whether the IOMMU is optional or mandatory would be a good enhancement.

---
 drivers/acpi/arm64/iort.c  | 25 ++++++++++++++++++++++++-
 drivers/acpi/scan.c        |  7 +++++--
 drivers/base/dma-mapping.c |  2 +-
 include/acpi/acpi_bus.h    |  2 +-
 include/linux/acpi.h       |  7 +++++--
 5 files changed, 36 insertions(+), 7 deletions(-)

index 3dd9ec3..a1000fb 100644
@@ -550,8 +550,17 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 			return NULL;
 
 		ops = iommu_ops_from_fwnode(iort_fwnode);
+		/*
+		 * If the ops look-up fails, this means that either
+		 * the SMMU drivers have not been probed yet or that
+		 * the SMMU drivers are not built in the kernel;
+		 * Depending on whether the SMMU drivers are built-in
+		 * in the kernel or not, defer the IOMMU configuration
+		 * or just abort it.
+		 */
 		if (!ops)
-			return NULL;
+			return iort_iommu_driver_enabled(node->type) ?
+			       ERR_PTR(-EPROBE_DEFER) : NULL;
 
 		ret = arm_smmu_iort_xlate(dev, streamid, iort_fwnode, ops);
 	}

@@ -625,12 +634,26 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 
 		while (parent) {
 			ops = iort_iommu_xlate(dev, parent, streamid);
+			if (IS_ERR_OR_NULL(ops))
+				return ops;
 
 			parent = iort_node_get_id(node, &streamid,
 						  IORT_IOMMU_TYPE, i++);
 		}
 	}
 
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
+
 	return ops;
 }
 

index 1926918..823b005 100644
@@ -1373,20 +1373,23 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
  * @dev: The pointer to the device
  * @attr: device dma attributes
  */
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
-
+	if (IS_ERR(iommu))
+		return PTR_ERR(iommu);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
 	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
 			   attr == DEV_DMA_COHERENT);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(acpi_dma_configure);
 

index 82bd45c..755a2b5 100644
@@ -368,7 +368,7 @@ int dma_configure(struct device *dev)
 	} else if (has_acpi_companion(dma_dev)) {
 		attr = acpi_get_dma_attr(to_acpi_device_node(dma_dev->fwnode));
 		if (attr != DEV_DMA_NOT_SUPPORTED)
-			acpi_dma_configure(dev, attr);
+			ret = acpi_dma_configure(dev, attr);
 	}
 
 	if (bridge)

index ef0ae8a..2a9a5de 100644
@@ -575,7 +575,7 @@ struct acpi_pci_root {
 
 bool acpi_dma_supported(struct acpi_device *adev);
 enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev);
-void acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
+int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr);
 void acpi_dma_deconfigure(struct device *dev);
 
 struct acpi_device *acpi_find_child_device(struct acpi_device *parent,

index 673acda..60f0aad 100644
@@ -765,8 +765,11 @@ static inline enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 	return DEV_DMA_NOT_SUPPORTED;
 }
 
-static inline void acpi_dma_configure(struct device *dev,
-				      enum dev_dma_attr attr) { }
+static inline int acpi_dma_configure(struct device *dev,
+				     enum dev_dma_attr attr)
+{
+	return 0;
+}
 
 static inline void acpi_dma_deconfigure(struct device *dev) { }
 

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

With arch_setup_dma_ops now being called late during device's probe after the device's iommu is probed, the notifier trick required to handle the early setup of dma_ops before the iommu group gets created is not required. So removing the notifier's here.

---
 arch/arm64/mm/dma-mapping.c | 142 ++++++--------------------------------------
 1 file changed, 18 insertions(+), 124 deletions(-)

index 81cdb2e..b465759 100644
@@ -813,34 +813,26 @@ static void __iommu_unmap_sg_attrs(struct device *dev,
 	.mapping_error = iommu_dma_mapping_error,
 };
 
-/*
- * TODO: Right now __iommu_setup_dma_ops() gets called too early to do
- * everything it needs to - the device is only partially created and the
- * IOMMU driver hasn't seen it yet, so it can't have a group. Thus we
- * need this delayed attachment dance. Once IOMMU probe ordering is sorted
- * to move the arch_setup_dma_ops() call later, all the notifier bits below
- * become unnecessary, and will go away.
- */
-struct iommu_dma_notifier_data {
-	struct list_head list;
-	struct device *dev;
-	const struct iommu_ops *ops;
-	u64 dma_base;
-	u64 size;
-};
-static LIST_HEAD(iommu_dma_masters);
-static DEFINE_MUTEX(iommu_dma_notifier_lock);
+static int __init __iommu_dma_init(void)
+{
+	return iommu_dma_init();
+}
+arch_initcall(__iommu_dma_init);
 
-static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			   u64 dma_base, u64 size)
+static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
+				  const struct iommu_ops *ops)
 {
-	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
+	struct iommu_domain *domain;
+
+	if (!ops)
+		return;
 
 	/*
-	 * If the IOMMU driver has the DMA domain support that we require,
-	 * then the IOMMU core will have already configured a group for this
-	 * device, and allocated the default domain for that group.
+	 * The IOMMU core code allocates the default DMA domain, which the
+	 * underlying IOMMU driver needs to support via the dma-iommu layer.
 	 */
+	domain = iommu_get_domain_for_dev(dev);
+
 	if (!domain)
 		goto out_err;
 

@@ -851,109 +843,11 @@ static bool do_iommu_attach(struct device *dev, const struct iommu_ops *ops,
 		dev->dma_ops = &iommu_dma_ops;
 	}
 
-	return true;
+	return;
+
 out_err:
-	pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
+	 pr_warn("Failed to set up IOMMU for device %s; retaining platform DMA ops\n",
 		 dev_name(dev));
-	return false;
-}
-
-static void queue_iommu_attach(struct device *dev, const struct iommu_ops *ops,
-			      u64 dma_base, u64 size)
-{
-	struct iommu_dma_notifier_data *iommudata;
-
-	iommudata = kzalloc(sizeof(*iommudata), GFP_KERNEL);
-	if (!iommudata)
-		return;
-
-	iommudata->dev = dev;
-	iommudata->ops = ops;
-	iommudata->dma_base = dma_base;
-	iommudata->size = size;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_add(&iommudata->list, &iommu_dma_masters);
-	mutex_unlock(&iommu_dma_notifier_lock);
-}
-
-static int __iommu_attach_notifier(struct notifier_block *nb,
-				   unsigned long action, void *data)
-{
-	struct iommu_dma_notifier_data *master, *tmp;
-
-	if (action != BUS_NOTIFY_BIND_DRIVER)
-		return 0;
-
-	mutex_lock(&iommu_dma_notifier_lock);
-	list_for_each_entry_safe(master, tmp, &iommu_dma_masters, list) {
-		if (data == master->dev && do_iommu_attach(master->dev,
-				master->ops, master->dma_base, master->size)) {
-			list_del(&master->list);
-			kfree(master);
-			break;
-		}
-	}
-	mutex_unlock(&iommu_dma_notifier_lock);
-	return 0;
-}
-
-static int __init register_iommu_dma_ops_notifier(struct bus_type *bus)
-{
-	struct notifier_block *nb = kzalloc(sizeof(*nb), GFP_KERNEL);
-	int ret;
-
-	if (!nb)
-		return -ENOMEM;
-
-	nb->notifier_call = __iommu_attach_notifier;
-
-	ret = bus_register_notifier(bus, nb);
-	if (ret) {
-		pr_warn("Failed to register DMA domain notifier; IOMMU DMA ops unavailable on bus '%s'\n",
-			bus->name);
-		kfree(nb);
-	}
-	return ret;
-}
-
-static int __init __iommu_dma_init(void)
-{
-	int ret;
-
-	ret = iommu_dma_init();
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&platform_bus_type);
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&amba_bustype);
-#ifdef CONFIG_PCI
-	if (!ret)
-		ret = register_iommu_dma_ops_notifier(&pci_bus_type);
-#endif
-	return ret;
-}
-arch_initcall(__iommu_dma_init);
-
-static void __iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size,
-				  const struct iommu_ops *ops)
-{
-	struct iommu_group *group;
-
-	if (!ops)
-		return;
-	/*
-	 * TODO: As a concession to the future, we're ready to handle being
-	 * called both early and late (i.e. after bus_add_device). Once all
-	 * the platform bus code is reworked to call us late and the notifier
-	 * junk above goes away, move the body of do_iommu_attach here.
-	 */
-	group = iommu_group_get(dev);
-	if (group) {
-		do_iommu_attach(dev, ops, dma_base, size);
-		iommu_group_put(group);
-	} else {
-		queue_iommu_attach(dev, ops, dma_base, size);
-	}
 }
 
 void arch_teardown_dma_ops(struct device *dev)

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

From: Robin Murphy <[email protected]>

Now that the appropriate ordering is enforced via probe-deferral of masters in core code, rip it all out and bask in the simplicity.

---
 drivers/iommu/arm-smmu-v3.c |  46 +-----------------
 drivers/iommu/arm-smmu.c    | 110 +++++++++++++++++++-------------------------
 2 files changed, 49 insertions(+), 107 deletions(-)

index 5806a6a..9497800 100644
@@ -2761,51 +2761,9 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
 	.probe	= arm_smmu_device_probe,
 	.remove	= arm_smmu_device_remove,
 };
+module_platform_driver(arm_smmu_driver);
 
-static int __init arm_smmu_init(void)
-{
-	static bool registered;
-	int ret = 0;
-
-	if (!registered) {
-		ret = platform_driver_register(&arm_smmu_driver);
-		registered = !ret;
-	}
-	return ret;
-}
-
-static void __exit arm_smmu_exit(void)
-{
-	return platform_driver_unregister(&arm_smmu_driver);
-}
-
-subsys_initcall(arm_smmu_init);
-module_exit(arm_smmu_exit);
-
-static int __init arm_smmu_of_init(struct device_node *np)
-{
-	int ret = arm_smmu_init();
-
-	if (ret)
-		return ret;
-
-	if (!of_platform_device_create(np, NULL, platform_bus_type.dev_root))
-		return -ENODEV;
-
-	return 0;
-}
-IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", arm_smmu_of_init);
-
-#ifdef CONFIG_ACPI
-static int __init acpi_smmu_v3_init(struct acpi_table_header *table)
-{
-	if (iort_node_match(ACPI_IORT_NODE_SMMU_V3))
-		return arm_smmu_init();
-
-	return 0;
-}
-IORT_ACPI_DECLARE(arm_smmu_v3, ACPI_SIG_IORT, acpi_smmu_v3_init);
-#endif
+IOMMU_OF_DECLARE(arm_smmuv3, "arm,smmu-v3", NULL);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations");
 MODULE_AUTHOR("Will Deacon <[email protected]>");

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

From: Lorenzo Pieralisi <[email protected]>

The IORT linker section introduced by commit 34ceea275f62 ("ACPI/IORT: Introduce linker section for IORT entries probing") was needed to make sure SMMU drivers are registered (and therefore probed) in the kernel before devices using the SMMU have a chance to probe in turn.

Through the introduction of deferred IOMMU configuration the linker section based IORT probing infrastructure is not needed any longer, in that device/SMMU probe dependencies are managed through the probe deferral mechanism, making the IORT linker section infrastructure unused, so that it can be removed.

Remove the unused IORT linker section probing infrastructure from the kernel to complete the ACPI IORT IOMMU configure probe deferral mechanism implementation.

---
 drivers/acpi/arm64/iort.c         | 2 --
 include/asm-generic/vmlinux.lds.h | 1 -
 include/linux/acpi_iort.h         | 3 ---
 3 files changed, 6 deletions(-)

index a1000fb..36a9abf 100644
@@ -992,6 +992,4 @@ void __init acpi_iort_init(void)
 	}
 
 	iort_init_platform_devices();
-
-	acpi_probe_device_table(iort);
 }

index 0968d13..9faa26c 100644
@@ -566,7 +566,6 @@
 	IRQCHIP_OF_MATCH_TABLE()					\
 	ACPI_PROBE_TABLE(irqchip)					\
 	ACPI_PROBE_TABLE(clksrc)					\
-	ACPI_PROBE_TABLE(iort)						\
 	EARLYCON_TABLE()
 
 #define INIT_TEXT							\

index 77e0809..f167e1d04 100644
@@ -52,7 +52,4 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 { return NULL; }
 #endif
 
-#define IORT_ACPI_DECLARE(name, table_id, fn)		\
-	ACPI_DECLARE_PROBE_ENTRY(iort, name, table_id, 0, NULL, 0, fn)
-
 #endif /* __ACPI_IORT_H__ */

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Hi Sricharan,

I applied this series on v4.11-rc1 to test PCIe pass through in HiSilicon D05 board(with Intel 82599 networking card). It failed.

After I used:

echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind echo 0002:81:10.0 > /sys/bus/pci/drivers_probe

to bind vfio-pci driver to Intel 82599 networking card VF.

I got log in host:
[...]

[  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k
[  414.275824] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs
[  414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
[  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI
[  414.343348] ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class 0x020000
[  414.448713] iommu: Adding device 0002:81:10.0 to group 4
[  414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002)
[  414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface up?
[  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address
[  414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02
[  414.451417] ixgbevf 0002:81:10.0: MAC: 1
[  414.451418] ixgbevf 0002:81:10.0: Intel(R) 82599 Virtual Function
[  414.464271] VFIO - User Level meta-driver version: 0.3
[  414.570074] ixgbe 0002:81:00.0: registered PHC device on eth12
[  414.700493] specified DMA range outside IOMMU capability                                      <-- error here
[  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining platform DMA ops        <-- error here
[  414.748043] ixgbe 0002:81:00.0 eth12: detected SFP+: 5
[  414.922277] ixgbe 0002:81:00.0 eth12: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Then I tried to boot up VM using:

qemu-system-aarch64 \
-machine virt,gic-version=3 \
-enable-kvm \
-cpu host \
-m 1024 \
-kernel ./Image_vm \
-initrd ./minifs.cpio.gz \
-nographic \
-net none -device vfio-pci,host=0002:81:10.0,id=net0

I got this error:

root@ubuntu:~/scripts# ./qemu_run.sh
[  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
[  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass vfio_virqfd vfio ixgbevf ixgb                                                                                                            e mdio [last unloaded: vfio_iommu_type1]
[  402.604733] CPU: 26 PID: 4437 Comm: qemu-system-aar Not tainted 4.11.0-rc1-g4b62e7fa #21
[  402.612809] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI 16.12 Release 02/22                                                                                                            /2017
[  402.622013] task: ffff8017e2161a00 task.stack: ffff8017e5a8c000
[  402.627926] PC is at arm_smmu_write_strtab_ent+0x1b4/0x1b8
[  402.633399] LR is at arm_smmu_install_ste_for_dev+0x9c/0xc0
[  402.638957] pc : [<ffff000008542b08>] lr : [<ffff000008542ba8>] pstate: 80000145
[  402.646338] sp : ffff8017e5a8fb50
[  402.649638] x29: ffff8017e5a8fb50 x28: ffff8013d3e4e468
[  402.654938] x27: ffff00000854240c x26: 0000ffffffffffff
[  402.660237] x25: ffff8013cd0b7100 x24: ffff8013e58fc018
[  402.665536] x23: ffff8013e58f8018 x22: 0000000000000018
[  402.670835] x21: 0000000000028180 x20: ffff8013e58f8018
[  402.676134] x19: 0000000000000001 x18: 0000000000783488
[  402.681434] x17: 0000ffff967caa50 x16: ffff000008207410
[  402.686733] x15: 0000000000782a54 x14: 0000ffff9670be14
[  402.692031] x13: 0000000000783060 x12: 0000000000000012
[  402.697330] x11: 0000000000000001 x10: 0000000000000900
[  402.702629] x9 : 00000000000000ff x8 : 0000000000000000
[  402.707928] x7 : ffff80003e040190 x6 : 000bdff385002ea1
[  402.713227] x5 : 0000000000000018 x4 : 0000000000000001
[  402.718526] x3 : ffff8013cd0b7108 x2 : ffff80003e6f6000
[  402.723824] x1 : ffff000008542ab0 x0 : ffff8013d3e4fe38
[  402.729123]
[  402.730602] Process qemu-system-aar (pid: 4437, stack limit = 0xffff8017e5a8c000)
[  402.738070] Stack: (0xffff8017e5a8fb50 to 0xffff8017e5a90000)
[  402.743802] fb40:                                   ffff8017e5a8fb90 ffff000008542ba8
[  402.751617] fb60: 0000000000000002 ffff8013cd0b7200 ffff8013cd0b7108 ffff8013e58f8140
[  402.759432] fb80: ffff8013e58f8018 ffff80003e6f6000 ffff8017e5a8fbd0 ffff000008543514
[  402.767247] fba0: ffff8013d3e4fe68 ffff8013cd0b7108 0000000000000000 ffff8013e68140a0
[  402.775063] fbc0: ffff8013d3e4fe08 ffff8013d3e4fe38 ffff8017e5a8fc80 ffff00000853a840
[  402.782878] fbe0: ffff8013e432df00 ffff8013d3e4fe68 ffff8013ccb28d48 ffff8013ccb28d00
[  402.790693] fc00: ffff8017e5f69a08 ffff8017e5f69a80 ffff00000096f380 ffff8017e5f69a18
[  402.798508] fc20: ffff8013ccb28d00 ffff8017e5f69a80 0000000000000000 0000000040201000
[  402.806323] fc40: 0000002c00000030 ffff0000089efae8 ffff8013e65b8410 00000017e59d8000
[  402.814138] fc60: 0000000000000000 0000000400803510 000000000004ff44 0000000000000000
[  402.821954] fc80: ffff8017e5a8fcb0 ffff00000853a8cc ffff8013ccb28d58 ffff8013d3e4fe68
[  402.829769] fca0: ffff8017e5f69b00 ffff0000009a93c0 ffff8017e5a8fce0 ffff0000009a7970
[  402.837584] fcc0: ffff8017e5f69b80 ffff8017e5f69a98 ffff8017e5a8fce0 ffff8013ccb28d00
[  402.845399] fce0: ffff8017e5a8fd90 ffff00000096c688 ffff8013d3e7f100 ffff8013ca9e9280
[  402.853214] fd00: ffff8017e5f69a00 0000000000000003 ffff8017e5f69a08 ffff8017e5f69a80
[  402.861029] fd20: ffff00000096f380 ffff8017e5f69a18 ffffffffffffffed ffff8017e2161a00
[  402.868844] fd40: ffff8017e5f69a08 0000000000000003 ffff00000096f380 ffff8017e5f69a18
[  402.876659] fd60: ffffffffffffffed 0000400000000010 ffff8017e5a8fd90 ffff000008e492d8
[  402.884475] fd80: 0000000000003b66 ffff8013ca9e9280 ffff8017e5a8fdf0 ffff000008206d28
[  402.892290] fda0: ffff8013d3e4fd00 0000000000000003 ffff8013d43a0318 0000000000000010
[  402.900105] fdc0: 0000000000003b66 0000000000000003 0000000000000123 000000000000001d
[  402.907920] fde0: ffff000008902000 0000000000000010 ffff8017e5a8fe80 ffff000008207494
[  402.915735] fe00: 0000000000000000 ffff8013d3e4fd01 ffff8013d3e4fd00 0000000000000010
[  402.923550] fe20: 0000000000003b66 ffff0000081f2594 ffff8017e5a8fe60 ffff000008212508
[  402.931365] fe40: ffff8017e5a8fe80 ffff000008207450 0000000000000000 ffff8013d3e4fd01
[  402.939180] fe60: ffff8013d3e4fd00 0000000000000010 0000000000003b66 ffff000008207434
[  402.946995] fe80: 0000000000000000 ffff000008082f30 0000000000000000 00008017f31b0000
[  402.954810] fea0: ffffffffffffffff 0000ffff967caa5c 0000000020000000 0000000000000015
[  402.962625] fec0: 0000000000000010 0000000000003b66 0000000000000003 0000000000000003
[  402.970440] fee0: 0000000000000000 0000000000000040 000000000000003f 0000000000000000
[  402.978255] ff00: 000000000000001d 0000000000000004 0000000000000000 0000ffffe6bbe7e0
[  402.986070] ff20: ffffffffffffffff 0000000000783060 0000ffff9670be14 0000000000782a54
[  402.993885] ff40: 0000000000a74630 0000ffff967caa50 0000000000783488 000000003151e080
[  403.001700] ff60: 0000ffffe6bbe86c 0000000000000004 0000000000af7000 0000000000b15000
[  403.009515] ff80: 0000000000af8450 0000000031222c50 0000000000000001 0000000031abb800
[  403.017331] ffa0: 0000000031de2650 0000ffffe6bbe7e0 00000000004b8468 0000ffffe6bbe7e0
[  403.025146] ffc0: 0000ffff967caa5c 0000000020000000 0000000000000010 000000000000001d
[  403.032961] ffe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[  403.040775] Call trace:
[  403.043208] Exception stack(0xffff8017e5a8f980 to 0xffff8017e5a8fab0)
[  403.049635] f980: 0000000000000001 0001000000000000 ffff8017e5a8fb50 ffff000008542b08
[  403.057450] f9a0: ffff000008e1b778 0000000000000001 ffff000008f1ebb8 00000000000006fc
[  403.065265] f9c0: 0000000000000000 000000000003e6fb 0000000400000123 000000000003e800
[  403.073080] f9e0: 0000000000000000 0000000000000000 ffff8017e5a8f9f0 ffff8017e5a8f9f0
[  403.080895] fa00: ffff8017e5a8fa50 ffff00000838feb0 ffff8013e58f8030 0000005d4ebe3be6
[  403.088711] fa20: ffff8013d3e4fe38 ffff000008542ab0 ffff80003e6f6000 ffff8013cd0b7108
[  403.096526] fa40: 0000000000000001 0000000000000018 000bdff385002ea1 ffff80003e040190
[  403.104341] fa60: 0000000000000000 00000000000000ff 0000000000000900 0000000000000001
[  403.112155] fa80: 0000000000000012 0000000000783060 0000ffff9670be14 0000000000782a54
[  403.119970] faa0: ffff000008207410 0000ffff967caa50
[  403.124835] [<ffff000008542b08>] arm_smmu_write_strtab_ent+0x1b4/0x1b8
[  403.131349] [<ffff000008542ba8>] arm_smmu_install_ste_for_dev+0x9c/0xc0
[  403.137950] [<ffff000008543514>] arm_smmu_attach_dev+0x1a4/0x2c0
[  403.143942] [<ffff00000853a840>] __iommu_attach_group+0x48/0xa8
[  403.149848] [<ffff00000853a8cc>] iommu_attach_group+0x2c/0x48
[  403.155584] [<ffff0000009a7970>] vfio_iommu_type1_attach_group+0x208/0x720 [vfio_iommu_type1]
[  403.164099] [<ffff00000096c688>] vfio_fops_unl_ioctl+0x188/0x2b8 [vfio]
[  403.170701] [<ffff000008206d28>] do_vfs_ioctl+0xb4/0x79c
[  403.175998] [<ffff000008207494>] SyS_ioctl+0x84/0x98
[  403.180950] [<ffff000008082f30>] el0_svc_naked+0x24/0x28
[  403.186249] Code: f9400860 b5fff9c0 17ffffd9 d4210000 (d4210000)
[  403.192351] ---[ end trace 0bda8b3549cfd903 ]---

Dropping this series, 82599 VF can work well in QEMU.

P.S.

To avoid some hardware bugs, I temporarily commented/added some codes in v4.11-rc1:

@@ -1104,8 +1104,8 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_device *smmu, u32 sid,
arm_smmu_sync_ste_for_sid(smmu, sid);

/* It's likely that we'll want to use the new STE soon */
- if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
- arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
+// if (!(smmu->options & ARM_SMMU_OPT_SKIP_PREFETCH))
+// arm_smmu_cmdq_issue_cmd(smmu, &prefetch_cmd);
}

static void arm_smmu_init_bypass_stes(u64 *strtab, unsigned int nent)

@@ -664,7 +664,7 @@ static void its_irq_compose_msi_msg(struct irq_data *d, struct msi_msg *msg)
msg->address_hi = upper_32_bits(addr);
msg->data = its_get_event_id(d);

- iommu_dma_map_msi_msg(d->irq, msg);
+ //iommu_dma_map_msi_msg(d->irq, msg);
}

  static struct irq_chip its_irq_chip = {

@@ -4179,6 +4179,14 @@ static int pci_quirk_qcom_rp_acs(struct pci_dev *dev, u16 acs_flags)
return ret;
}

+static int pci_quirk_hisi_rp_acs(struct pci_dev *dev, u16 acs_flags) +{
+ u16 flags = (PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF | PCI_ACS_SV);
+ int ret = acs_flags & ~flags ? 0 : 1;
+
+ return ret;
+}
+
/*
* Sunrise Point PCH root ports implement ACS, but unfortunately as shown in
* the datasheet (Intel 100 Series Chipset Family PCH Datasheet, Vol. 2,
@@ -4345,6 +4353,8 @@ static const struct pci_dev_acs_enabled {
{ 0x10df, 0x720, pci_quirk_mf_endpoint_acs }, /* Emulex Skyhawk-R */ /* Cavium ThunderX */
{ PCI_VENDOR_ID_CAVIUM, PCI_ANY_ID, pci_quirk_cavium_acs },
+ /* Hisilicon Hip05/Hip06 root ports */
+ { PCI_VENDOR_ID_HUAWEI, 0x1610, pci_quirk_hisi_rp_acs },
{ 0 }
};

If you have time, could you please take a look at this issue?

Thanks,
Zhou

Hi Zhou,

Looks like this triggers the start of the bug. So the below check in iommu_dma_init_domain fails,

        if (domain->geometry.force_aperture) {
                if (base > domain->geometry.aperture_end ||
                    base + size <= domain->geometry.aperture_start) {

and the rest goes out of sync after that. Can you print out the base, aperture_start and end values to see why the check fails ?

The change that this series does is trying to add the dma/iommu ops to the device after the iommu is actually probed. So in your working case, does the device initially gets hooked to iommu_ops and the above same check passes in working case ?

Regards,
Sricharan

"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Hi Sricharan,

dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size, domain->geometry.aperture_start, domain->geometry.aperture_end, *dev->dma_mask, dev->coherent_dma_mask);

[  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
.....
[  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff

Yes, size seems to be the problem here. When the VF device gets attached to vfio-pci, somehow the dev->coherent_dma_mask is set to 64 bits and size become zero.

@@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
if (ret < 0) {
dma_addr = offset = 0;
- size = dev->coherent_dma_mask + 1;
+ size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);

@@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
* Assume dma valid range starts at 0 and covers the whole
* coherent_dma_mask.
*/
- arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
+ size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
+ arch_setup_dma_ops(dev, 0, size, iommu,
attr == DEV_DMA_COHERENT);

With the above fixes, DT boot works fine. But we still get the below crash on ACPI

I believe so. Because didn't notice the "specified DMA range outside IOMMU capability" in the working case.

Thanks,
Shameer

Hi Shameer,

Looks like this happens when the ste_live becomes true during the initial attach, but later without an detach, attach once again happens from vfio. Just thinking why the detach_dev is not happening only in ACPI case. Actually not having any of arm-smmuv3 or ACPI based setup in my place. Can i get some help by having some logs from the arm-smmv3 driver ?

ok, as i said above not sure why the overflow does not affect without this series.

Regards,
Sricharan

"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

On Fri, Mar 24, 2017 at 09:27:51AM +0000, Shameerali Kolothum Thodi wrote:

[...]

Sure. We will rerun the tests to create the logs again and share.

Also need to double check, v9 patches are applied correctly on top of our branch. If I remember correctly v7 was fine with DT(didn't test with ACPI). But then as Shicharan said it had that size fix.

Thanks,
Shameer
[...]

AFAICS, this is either down to patch 3 (which should apply on its own easily enough for testing), or patch 6, implying that somehow the vfio-pci device gets its DMA mask widened to 64 bits somewhere between very soon after after creation (where we originally called of_dma_configure()) and immediately before probe (where we now call it).

Either way I guess this is yet more motivation to write that "change the arch_setup_dma_ops() interface to take a mask instead of a size" patch...

I can't see how ACPI vs. DT would make any difference to the domain attach/detach mechanics getting into an invalid state (if the DT case works then unbinding the ixgbevf driver clearly does release the Stream ID correctly in general), so I'm more inclined to believe that something goes wrong with the fwspec in the ACPI path such that we end up touching the wrong STE altogether.

Robin.

[...]

Just applying the patch 3 and binding the device into vfio-pci is fine. Please find the log below (with dev_info debug added to iommu_dma_init_domain ).
...

[  142.851906] iommu: Adding device 0000:81:10.0 to group 6
[  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff   ---->dev_info()
[  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
[  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
[  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
[  142.865692] ixgbevf 0000:81:10.0: MAC: 1
[  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[  152.902894] nfs: server 172.18.45.166 not responding, still trying
[  188.980933] nfs: server 172.18.45.166 not responding, still trying
[  188.981298] nfs: server 172.18.45.166 OK
[  188.981593] nfs: server 172.18.45.166 OK
[  221.755626] VFIO - User Level meta-driver version: 0.3

...

Applied up to patch 6, and the issue appeared,

[  145.212351] iommu: Adding device 0000:81:10.0 to group 5
[  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
[  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
[  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
[  145.225944] ixgbevf 0000:81:10.0: MAC: 1
[  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[  154.947742] nfs: server 172.18.45.166 not responding, still trying
[  191.025780] nfs: server 172.18.45.166 not responding, still trying
[  191.026122] nfs: server 172.18.45.166 OK
[  191.026317] nfs: server 172.18.45.166 OK
[  263.706402] VFIO - User Level meta-driver version: 0.3
[  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
[  269.757617] specified DMA range outside IOMMU capability
[  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining platform DMA ops

From the logs its clear that when ixgbevf driver originally probes and adds the device to smmu the dma mask is 32, but when it binds to vfio-pci, it becomes 64 bit.

Also It looks like the iommu_dma_init_domain fn was not invoked in the first case when the vfio-pci binds the device.

I am attaching the full logs for both cases.

Ok. We will debug this separately and update.

Thanks,
Shameer

[    0.000000] Booting Linux on physical CPU 0x10000
[    0.000000] Linux version 4.11.0-rc1-g74f9bd0-dirty (shameer@shameer-ubuntu) (gcc version 4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 - Linaro GCC 4.9-2014.09) ) #156 SMP PREEMPT Mon Mar 27 13:13:50 BST 2017
[    0.000000] Boot CPU: AArch64 Processor [410fd082]
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: EFI v2.60 by EDK II
[    0.000000] efi:  SMBIOS=0x3f050000  SMBIOS 3.0=0x39b00000  ACPI=0x39bd0000  ACPI 2.0=0x39bd0014  MEMATTR=0x3c1b7018 
[    0.000000] cma: Reserved 16 MiB at 0x000000003e000000
[    0.000000] NUMA: Adding memblock [0x0 - 0x3fffffff] on node 0
[    0.000000] OF: NUMA: parsing numa-distance-map-v1
[    0.000000] NUMA: Warning: invalid memblk node 4 [mem 0x1040000000-0x17fbffffff]
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x00000017fbffffff]
[    0.000000] NUMA: Adding memblock [0x0 - 0x257ff] on node 0
[    0.000000] NUMA: Adding memblock [0x25800 - 0x3187ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x31880000 - 0x3188ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x31890000 - 0x318fafff] on node 0
[    0.000000] NUMA: Adding memblock [0x318fb000 - 0x318fefff] on node 0
[    0.000000] NUMA: Adding memblock [0x318ff000 - 0x3999ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x399a0000 - 0x3a05ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x3a060000 - 0x3f04ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x3f050000 - 0x3f07ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x3f080000 - 0x3fbfffff] on node 0
[    0.000000] NUMA: Adding memblock [0x1040000000 - 0x17fbffffff] on node 0
[    0.000000] NUMA: Initmem setup node 0 [mem 0x00000000-0x17fbffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x17fbfc4d00-0x17fbfc67ff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000017fbffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000000024fff]
[    0.000000]   node   0: [mem 0x0000000000026000-0x000000003187ffff]
[    0.000000]   node   0: [mem 0x0000000031880000-0x000000003188ffff]
[    0.000000]   node   0: [mem 0x0000000031890000-0x00000000318fafff]
[    0.000000]   node   0: [mem 0x00000000318fb000-0x00000000318fefff]
[    0.000000]   node   0: [mem 0x00000000318ff000-0x000000003999ffff]
[    0.000000]   node   0: [mem 0x00000000399a0000-0x000000003a05ffff]
[    0.000000]   node   0: [mem 0x000000003a060000-0x000000003f04ffff]
[    0.000000]   node   0: [mem 0x000000003f050000-0x000000003f07ffff]
[    0.000000]   node   0: [mem 0x000000003f080000-0x000000003fbfffff]
[    0.000000]   node   0: [mem 0x0000001040000000-0x00000017fbffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000017fbffffff]
[    0.000000] On node 0 totalpages: 8371199
[    0.000000]   DMA zone: 4080 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 261119 pages, LIFO batch:31
[    0.000000]   Normal zone: 126720 pages used for memmap
[    0.000000]   Normal zone: 8110080 pages, LIFO batch:31
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] percpu: Embedded 23 pages/cpu @ffff8017db640000 s56960 r8192 d29056 u94208
[    0.000000] pcpu-alloc: s56960 r8192 d29056 u94208 alloc=23*4096
[    0.000000] pcpu-alloc: [0] 00 [0] 01 [0] 02 [0] 03 [0] 04 [0] 05 [0] 06 [0] 07 
[    0.000000] pcpu-alloc: [0] 08 [0] 09 [0] 10 [0] 11 [0] 12 [0] 13 [0] 14 [0] 15 
[    0.000000] pcpu-alloc: [0] 16 [0] 17 [0] 18 [0] 19 [0] 20 [0] 21 [0] 22 [0] 23 
[    0.000000] pcpu-alloc: [0] 24 [0] 25 [0] 26 [0] 27 [0] 28 [0] 29 [0] 30 [0] 31 
[    0.000000] pcpu-alloc: [0] 32 [0] 33 [0] 34 [0] 35 [0] 36 [0] 37 [0] 38 [0] 39 
[    0.000000] pcpu-alloc: [0] 40 [0] 41 [0] 42 [0] 43 [0] 44 [0] 45 [0] 46 [0] 47 
[    0.000000] pcpu-alloc: [0] 48 [0] 49 [0] 50 [0] 51 [0] 52 [0] 53 [0] 54 [0] 55 
[    0.000000] pcpu-alloc: [0] 56 [0] 57 [0] 58 [0] 59 [0] 60 [0] 61 [0] 62 [0] 63 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 8240399
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/shameer/Image_smmu_4.11 rdinit=/init console=ttyAMA0,115200 earlycon=sbsa-uart,mmio,0x602b0000 pcie_aspm=off acpi=off noinitrd root=/dev/nfs rw nfsroot=172.18.45.166:/home/shameer/sherlok-files/ubuntu-sher/ubuntu,nfsvers=3 ip=dhcp::eth4:on:172.18.45.1:8.8.8.8 user_debug=1 nfsrootdebug pcie_acs_override=downstream
[    0.000000] PCIe ASPM is disabled
[    0.000000] log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] log_buf_len total cpu_extra contributions: 258048 bytes
[    0.000000] log_buf_len min size: 131072 bytes
[    0.000000] log_buf_len: 524288 bytes
[    0.000000] early log buf free: 125900(96%)
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] software IO TLB [mem 0x359a0000-0x399a0000] (64MB) mapped at [ffff8000359a0000-ffff80003999ffff]
[    0.000000] Memory: 32848756K/33484796K available (8828K kernel code, 964K rwdata, 3976K rodata, 1088K init, 398K bss, 619656K reserved, 16384K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000   (   128 MB)
[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000   (129022 GB)
[    0.000000]       .text : 0xffff000008080000 - 0xffff000008920000   (  8832 KB)
[    0.000000]     .rodata : 0xffff000008920000 - 0xffff000008d10000   (  4032 KB)
[    0.000000]       .init : 0xffff000008d10000 - 0xffff000008e20000   (  1088 KB)
[    0.000000]       .data : 0xffff000008e20000 - 0xffff000008f11200   (   965 KB)
[    0.000000]        .bss : 0xffff000008f11200 - 0xffff000008f74db8   (   399 KB)
[    0.000000]     fixed   : 0xffff7dfffe7fd000 - 0xffff7dfffec00000   (  4108 KB)
[    0.000000]     PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000   (    16 MB)
[    0.000000]     vmemmap : 0xffff7e0000000000 - 0xffff800000000000   (  2048 GB maximum)
[    0.000000]               0xffff7e0000000000 - 0xffff7e005ff00000   (  1535 MB actual)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff8017fc000000   ( 98240 MB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] NR_IRQS:64 nr_irqs:64 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] ITS [mem 0x4c000000-0x4c03ffff]
[    0.000000] ITS@0x000000004c000000: allocated 524288 Devices @17da800000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000000004c000000: allocated 2048 Virtual CPUs @17db024000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000000004c000000: allocated 512 Interrupt Collections @17db021000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x6c000000-0x6c03ffff]
[    0.000000] ITS@0x000000006c000000: allocated 524288 Devices @17da400000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000000006c000000: allocated 2048 Virtual CPUs @17db028000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000000006c000000: allocated 512 Interrupt Collections @17db023000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0xc6000000-0xc603ffff]
[    0.000000] ITS@0x00000000c6000000: allocated 524288 Devices @17da000000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000000c6000000: allocated 2048 Virtual CPUs @17db02c000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000000c6000000: allocated 512 Interrupt Collections @17db060000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x8c6000000-0x8c603ffff]
[    0.000000] ITS@0x00000008c6000000: allocated 524288 Devices @17d9c00000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000008c6000000: allocated 2048 Virtual CPUs @17db064000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000008c6000000: allocated 512 Interrupt Collections @17db061000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x4004c000000-0x4004c03ffff]
[    0.000000] ITS@0x000004004c000000: allocated 524288 Devices @17d9800000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000004004c000000: allocated 2048 Virtual CPUs @17db068000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000004004c000000: allocated 512 Interrupt Collections @17db062000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x4006c000000-0x4006c03ffff]
[    0.000000] ITS@0x000004006c000000: allocated 524288 Devices @17d9400000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000004006c000000: allocated 2048 Virtual CPUs @17db06c000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000004006c000000: allocated 512 Interrupt Collections @17db0a0000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x400c6000000-0x400c603ffff]
[    0.000000] ITS@0x00000400c6000000: allocated 524288 Devices @17d9000000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000400c6000000: allocated 2048 Virtual CPUs @17db0a4000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000400c6000000: allocated 512 Interrupt Collections @17db0a1000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x408c6000000-0x408c603ffff]
[    0.000000] ITS@0x00000408c6000000: allocated 524288 Devices @17d8c00000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000408c6000000: allocated 2048 Virtual CPUs @17db0a8000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000408c6000000: allocated 512 Interrupt Collections @17db0a2000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] GIC: using LPI property table @0x00000017db0d0000
[    0.000000] ITS: Allocated 1792 chunks for LPIs
[    0.000000] GICv3: CPU0: found redistributor 10000 region 0:0x000000004d100000
[    0.000000] CPU0: using LPI pending table @0x00000017db0e0000
[    0.000000] arm_arch_timer: Architected cp15 timer(s) running at 50.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
[    0.000001] sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
[    0.000105] Console: colour dummy device 80x25
[    0.000137] Calibrating delay loop (skipped), value calculated using timer frequency.. 100.00 BogoMIPS (lpj=200000)
[    0.000141] pid_max: default: 65536 minimum: 512
[    0.000184] Security Framework initialized
[    0.003838] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
[    0.013302] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.017183] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.017242] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.043989] ASID allocator initialised with 65536 entries
[    0.064017] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@4c000000 domain created
[    0.064021] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@6c000000 domain created
[    0.064024] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@c6000000 domain created
[    0.064027] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@8,c6000000 domain created
[    0.064031] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@400,4c000000 domain created
[    0.064034] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@400,6c000000 domain created
[    0.064037] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@400,c6000000 domain created
[    0.064040] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@408,c6000000 domain created
[    0.064066] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@4c000000 domain created
[    0.064069] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@6c000000 domain created
[    0.064072] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@c6000000 domain created
[    0.064075] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@8,c6000000 domain created
[    0.064078] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@400,4c000000 domain created
[    0.064081] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@400,6c000000 domain created
[    0.064084] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@400,c6000000 domain created
[    0.064087] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@408,c6000000 domain created
[    0.064273] Remapping and enabling EFI services.
[    0.064309]   EFI remap 0x00000000399a0000 => 0000000020000000
[    0.064330]   EFI remap 0x00000000399f0000 => 0000000020050000
[    0.064339]   EFI remap 0x0000000039a40000 => 00000000200a0000
[    0.064370]   EFI remap 0x0000000039af0000 => 00000000200b0000
[    0.064392]   EFI remap 0x0000000039b70000 => 0000000020130000
[    0.064413]   EFI remap 0x0000000039be0000 => 0000000020180000
[    0.064439]   EFI remap 0x0000000039c30000 => 00000000201d0000
[    0.064459]   EFI remap 0x0000000039c80000 => 0000000020220000
[    0.064481]   EFI remap 0x0000000039cd0000 => 0000000020270000
[    0.064502]   EFI remap 0x0000000039d20000 => 00000000202c0000
[    0.064524]   EFI remap 0x0000000039d70000 => 0000000020310000
[    0.064546]   EFI remap 0x0000000039dc0000 => 0000000020360000
[    0.064566]   EFI remap 0x0000000039e10000 => 00000000203b0000
[    0.064597]   EFI remap 0x0000000039e60000 => 0000000020400000
[    0.064618]   EFI remap 0x0000000039ed0000 => 0000000020470000
[    0.064639]   EFI remap 0x0000000039f20000 => 00000000204c0000
[    0.064661]   EFI remap 0x0000000039f70000 => 0000000020510000
[    0.064683]   EFI remap 0x0000000039fc0000 => 0000000020560000
[    0.064704]   EFI remap 0x000000003a010000 => 00000000205b0000
[    0.064722]   EFI remap 0x000000003f050000 => 0000000020600000
[    0.064731]   EFI remap 0x0000000078000000 => 0000000020630000
[    0.064737]   EFI remap 0x00000000a4000000 => 0000000020800000
[    0.064749]   EFI remap 0x00000000a6000000 => 0000000021800000
[    0.064758]   EFI remap 0x00000000d00e0000 => 0000000021810000
[    0.080009] smp: Bringing up secondary CPUs ...
[    0.162612] Detected PIPT I-cache on CPU1
[    0.162623] GICv3: CPU1: found redistributor 10001 region 0:0x000000004d140000
[    0.162646] CPU1: using LPI pending table @0x00000017d5850000
[    0.162793] CPU1: Booted secondary processor [410fd082]
[    0.245232] Detected PIPT I-cache on CPU2
[    0.245239] GICv3: CPU2: found redistributor 10002 region 0:0x000000004d180000
[    0.245261] CPU2: using LPI pending table @0x00000017d5880000
[    0.245401] CPU2: Booted secondary processor [410fd082]
[    0.327854] Detected PIPT I-cache on CPU3
[    0.327859] GICv3: CPU3: found redistributor 10003 region 0:0x000000004d1c0000
[    0.327882] CPU3: using LPI pending table @0x00000017d58c0000
[    0.328023] CPU3: Booted secondary processor [410fd082]
[    0.410479] Detected PIPT I-cache on CPU4
[    0.410490] GICv3: CPU4: found redistributor 10100 region 0:0x000000004d200000
[    0.410513] CPU4: using LPI pending table @0x00000017d58f0000
[    0.410655] CPU4: Booted secondary processor [410fd082]
[    0.493101] Detected PIPT I-cache on CPU5
[    0.493108] GICv3: CPU5: found redistributor 10101 region 0:0x000000004d240000
[    0.493129] CPU5: using LPI pending table @0x00000017d5920000
[    0.493269] CPU5: Booted secondary processor [410fd082]
[    0.575725] Detected PIPT I-cache on CPU6
[    0.575732] GICv3: CPU6: found redistributor 10102 region 0:0x000000004d280000
[    0.575755] CPU6: using LPI pending table @0x00000017d5950000
[    0.575895] CPU6: Booted secondary processor [410fd082]
[    0.658350] Detected PIPT I-cache on CPU7
[    0.658356] GICv3: CPU7: found redistributor 10103 region 0:0x000000004d2c0000
[    0.658379] CPU7: using LPI pending table @0x00000017d5980000
[    0.658520] CPU7: Booted secondary processor [410fd082]
[    0.740977] Detected PIPT I-cache on CPU8
[    0.740988] GICv3: CPU8: found redistributor 10200 region 0:0x000000004d300000
[    0.741011] CPU8: using LPI pending table @0x00000017d59b0000
[    0.741152] CPU8: Booted secondary processor [410fd082]
[    0.823600] Detected PIPT I-cache on CPU9
[    0.823606] GICv3: CPU9: found redistributor 10201 region 0:0x000000004d340000
[    0.823628] CPU9: using LPI pending table @0x00000017d59f0000
[    0.823767] CPU9: Booted secondary processor [410fd082]
[    0.906224] Detected PIPT I-cache on CPU10
[    0.906231] GICv3: CPU10: found redistributor 10202 region 0:0x000000004d380000
[    0.906253] CPU10: using LPI pending table @0x00000017d5a20000
[    0.906393] CPU10: Booted secondary processor [410fd082]
[    0.988849] Detected PIPT I-cache on CPU11
[    0.988855] GICv3: CPU11: found redistributor 10203 region 0:0x000000004d3c0000
[    0.988878] CPU11: using LPI pending table @0x00000017d5a70000
[    0.989108] CPU11: Booted secondary processor [410fd082]
[    1.071477] Detected PIPT I-cache on CPU12
[    1.071488] GICv3: CPU12: found redistributor 10300 region 0:0x000000004d400000
[    1.071511] CPU12: using LPI pending table @0x00000017d5aa0000
[    1.071654] CPU12: Booted secondary processor [410fd082]
[    1.154099] Detected PIPT I-cache on CPU13
[    1.154106] GICv3: CPU13: found redistributor 10301 region 0:0x000000004d440000
[    1.154129] CPU13: using LPI pending table @0x00000017d5ad0000
[    1.154267] CPU13: Booted secondary processor [410fd082]
[    1.236724] Detected PIPT I-cache on CPU14
[    1.236731] GICv3: CPU14: found redistributor 10302 region 0:0x000000004d480000
[    1.236753] CPU14: using LPI pending table @0x00000017d5b00000
[    1.236894] CPU14: Booted secondary processor [410fd082]
[    1.319348] Detected PIPT I-cache on CPU15
[    1.319355] GICv3: CPU15: found redistributor 10303 region 0:0x000000004d4c0000
[    1.319378] CPU15: using LPI pending table @0x00000017d5b40000
[    1.319519] CPU15: Booted secondary processor [410fd082]
[    1.401626] Detected PIPT I-cache on CPU16
[    1.401635] GICv3: CPU16: found redistributor 30000 region 1:0x000000006d100000
[    1.401652] CPU16: using LPI pending table @0x00000017d5b70000
[    1.401758] CPU16: Booted secondary processor [410fd082]
[    1.484254] Detected PIPT I-cache on CPU17
[    1.484261] GICv3: CPU17: found redistributor 30001 region 1:0x000000006d140000
[    1.484278] CPU17: using LPI pending table @0x00000017d5b90000
[    1.484384] CPU17: Booted secondary processor [410fd082]
[    1.566884] Detected PIPT I-cache on CPU18
[    1.566891] GICv3: CPU18: found redistributor 30002 region 1:0x000000006d180000
[    1.566910] CPU18: using LPI pending table @0x00000017d5bd0000
[    1.567017] CPU18: Booted secondary processor [410fd082]
[    1.649513] Detected PIPT I-cache on CPU19
[    1.649520] GICv3: CPU19: found redistributor 30003 region 1:0x000000006d1c0000
[    1.649537] CPU19: using LPI pending table @0x00000017d5400000
[    1.649643] CPU19: Booted secondary processor [410fd082]
[    1.732144] Detected PIPT I-cache on CPU20
[    1.732153] GICv3: CPU20: found redistributor 30100 region 1:0x000000006d200000
[    1.732170] CPU20: using LPI pending table @0x00000017d5430000
[    1.732277] CPU20: Booted secondary processor [410fd082]
[    1.814771] Detected PIPT I-cache on CPU21
[    1.814779] GICv3: CPU21: found redistributor 30101 region 1:0x000000006d240000
[    1.814796] CPU21: using LPI pending table @0x00000017d5460000
[    1.814901] CPU21: Booted secondary processor [410fd082]
[    1.897401] Detected PIPT I-cache on CPU22
[    1.897408] GICv3: CPU22: found redistributor 30102 region 1:0x000000006d280000
[    1.897425] CPU22: using LPI pending table @0x00000017d5490000
[    1.897529] CPU22: Booted secondary processor [410fd082]
[    1.980030] Detected PIPT I-cache on CPU23
[    1.980038] GICv3: CPU23: found redistributor 30103 region 1:0x000000006d2c0000
[    1.980055] CPU23: using LPI pending table @0x00000017d54c0000
[    1.980161] CPU23: Booted secondary processor [410fd082]
[    2.062661] Detected PIPT I-cache on CPU24
[    2.062671] GICv3: CPU24: found redistributor 30200 region 1:0x000000006d300000
[    2.062688] CPU24: using LPI pending table @0x00000017d54f0000
[    2.062794] CPU24: Booted secondary processor [410fd082]
[    2.145288] Detected PIPT I-cache on CPU25
[    2.145296] GICv3: CPU25: found redistributor 30201 region 1:0x000000006d340000
[    2.145313] CPU25: using LPI pending table @0x00000017d5530000
[    2.145419] CPU25: Booted secondary processor [410fd082]
[    2.227917] Detected PIPT I-cache on CPU26
[    2.227925] GICv3: CPU26: found redistributor 30202 region 1:0x000000006d380000
[    2.227943] CPU26: using LPI pending table @0x00000017d5560000
[    2.228047] CPU26: Booted secondary processor [410fd082]
[    2.310547] Detected PIPT I-cache on CPU27
[    2.310555] GICv3: CPU27: found redistributor 30203 region 1:0x000000006d3c0000
[    2.310572] CPU27: using LPI pending table @0x00000017d5590000
[    2.310679] CPU27: Booted secondary processor [410fd082]
[    2.393177] Detected PIPT I-cache on CPU28
[    2.393187] GICv3: CPU28: found redistributor 30300 region 1:0x000000006d400000
[    2.393205] CPU28: using LPI pending table @0x00000017d55c0000
[    2.393309] CPU28: Booted secondary processor [410fd082]
[    2.475805] Detected PIPT I-cache on CPU29
[    2.475813] GICv3: CPU29: found redistributor 30301 region 1:0x000000006d440000
[    2.475830] CPU29: using LPI pending table @0x00000017d55f0000
[    2.475935] CPU29: Booted secondary processor [410fd082]
[    2.558433] Detected PIPT I-cache on CPU30
[    2.558441] GICv3: CPU30: found redistributor 30302 region 1:0x000000006d480000
[    2.558458] CPU30: using LPI pending table @0x00000017d5630000
[    2.558562] CPU30: Booted secondary processor [410fd082]
[    2.641062] Detected PIPT I-cache on CPU31
[    2.641070] GICv3: CPU31: found redistributor 30303 region 1:0x000000006d4c0000
[    2.641087] CPU31: using LPI pending table @0x00000017d5660000
[    2.641192] CPU31: Booted secondary processor [410fd082]
[    2.723375] Detected PIPT I-cache on CPU32
[    2.723411] GICv3: CPU32: found redistributor 50000 region 2:0x000004004d100000
[    2.723462] CPU32: using LPI pending table @0x00000017d5690000
[    2.723612] CPU32: Booted secondary processor [410fd082]
[    2.805997] Detected PIPT I-cache on CPU33
[    2.806014] GICv3: CPU33: found redistributor 50001 region 2:0x000004004d140000
[    2.806055] CPU33: using LPI pending table @0x00000017d56c0000
[    2.806160] CPU33: Booted secondary processor [410fd082]
[    2.888634] Detected PIPT I-cache on CPU34
[    2.888650] GICv3: CPU34: found redistributor 50002 region 2:0x000004004d180000
[    2.888693] CPU34: using LPI pending table @0x00000017d56f0000
[    2.888799] CPU34: Booted secondary processor [410fd082]
[    2.971274] Detected PIPT I-cache on CPU35
[    2.971290] GICv3: CPU35: found redistributor 50003 region 2:0x000004004d1c0000
[    2.971333] CPU35: using LPI pending table @0x00000017d5720000
[    2.971438] CPU35: Booted secondary processor [410fd082]
[    3.053915] Detected PIPT I-cache on CPU36
[    3.053932] GICv3: CPU36: found redistributor 50100 region 2:0x000004004d200000
[    3.053975] CPU36: using LPI pending table @0x00000017d5760000
[    3.054082] CPU36: Booted secondary processor [410fd082]
[    3.136554] Detected PIPT I-cache on CPU37
[    3.136570] GICv3: CPU37: found redistributor 50101 region 2:0x000004004d240000
[    3.136613] CPU37: using LPI pending table @0x00000017d5790000
[    3.136717] CPU37: Booted secondary processor [410fd082]
[    3.219195] Detected PIPT I-cache on CPU38
[    3.219211] GICv3: CPU38: found redistributor 50102 region 2:0x000004004d280000
[    3.219254] CPU38: using LPI pending table @0x00000017d57c0000
[    3.219360] CPU38: Booted secondary processor [410fd082]
[    3.301835] Detected PIPT I-cache on CPU39
[    3.301852] GICv3: CPU39: found redistributor 50103 region 2:0x000004004d2c0000
[    3.301895] CPU39: using LPI pending table @0x00000017d57f0000
[    3.302000] CPU39: Booted secondary processor [410fd082]
[    3.384477] Detected PIPT I-cache on CPU40
[    3.384494] GICv3: CPU40: found redistributor 50200 region 2:0x000004004d300000
[    3.384538] CPU40: using LPI pending table @0x00000017d5020000
[    3.384646] CPU40: Booted secondary processor [410fd082]
[    3.467115] Detected PIPT I-cache on CPU41
[    3.467131] GICv3: CPU41: found redistributor 50201 region 2:0x000004004d340000
[    3.467174] CPU41: using LPI pending table @0x00000017d5050000
[    3.467279] CPU41: Booted secondary processor [410fd082]
[    3.549753] Detected PIPT I-cache on CPU42
[    3.549770] GICv3: CPU42: found redistributor 50202 region 2:0x000004004d380000
[    3.549815] CPU42: using LPI pending table @0x00000017d5080000
[    3.549921] CPU42: Booted secondary processor [410fd082]
[    3.632394] Detected PIPT I-cache on CPU43
[    3.632410] GICv3: CPU43: found redistributor 50203 region 2:0x000004004d3c0000
[    3.632454] CPU43: using LPI pending table @0x00000017d50c0000
[    3.632559] CPU43: Booted secondary processor [410fd082]
[    3.715034] Detected PIPT I-cache on CPU44
[    3.715052] GICv3: CPU44: found redistributor 50300 region 2:0x000004004d400000
[    3.715097] CPU44: using LPI pending table @0x00000017d50e0000
[    3.715204] CPU44: Booted secondary processor [410fd082]
[    3.797673] Detected PIPT I-cache on CPU45
[    3.797690] GICv3: CPU45: found redistributor 50301 region 2:0x000004004d440000
[    3.797732] CPU45: using LPI pending table @0x00000017d5120000
[    3.797836] CPU45: Booted secondary processor [410fd082]
[    3.880315] Detected PIPT I-cache on CPU46
[    3.880331] GICv3: CPU46: found redistributor 50302 region 2:0x000004004d480000
[    3.880374] CPU46: using LPI pending table @0x00000017d5150000
[    3.880481] CPU46: Booted secondary processor [410fd082]
[    3.962953] Detected PIPT I-cache on CPU47
[    3.962970] GICv3: CPU47: found redistributor 50303 region 2:0x000004004d4c0000
[    3.963014] CPU47: using LPI pending table @0x00000017d5180000
[    3.963119] CPU47: Booted secondary processor [410fd082]
[    4.045250] Detected PIPT I-cache on CPU48
[    4.045273] GICv3: CPU48: found redistributor 70000 region 3:0x000004006d100000
[    4.045318] CPU48: using LPI pending table @0x00000017d51b0000
[    4.045465] CPU48: Booted secondary processor [410fd082]
[    4.127889] Detected PIPT I-cache on CPU49
[    4.127908] GICv3: CPU49: found redistributor 70001 region 3:0x000004006d140000
[    4.127953] CPU49: using LPI pending table @0x00000017d51e0000
[    4.128097] CPU49: Booted secondary processor [410fd082]
[    4.210530] Detected PIPT I-cache on CPU50
[    4.210550] GICv3: CPU50: found redistributor 70002 region 3:0x000004006d180000
[    4.210595] CPU50: using LPI pending table @0x00000017d5210000
[    4.210741] CPU50: Booted secondary processor [410fd082]
[    4.293170] Detected PIPT I-cache on CPU51
[    4.293189] GICv3: CPU51: found redistributor 70003 region 3:0x000004006d1c0000
[    4.293235] CPU51: using LPI pending table @0x00000017d5250000
[    4.293378] CPU51: Booted secondary processor [410fd082]
[    4.375814] Detected PIPT I-cache on CPU52
[    4.375836] GICv3: CPU52: found redistributor 70100 region 3:0x000004006d200000
[    4.375885] CPU52: using LPI pending table @0x00000017d5280000
[    4.376033] CPU52: Booted secondary processor [410fd082]
[    4.458453] Detected PIPT I-cache on CPU53
[    4.458472] GICv3: CPU53: found redistributor 70101 region 3:0x000004006d240000
[    4.458519] CPU53: using LPI pending table @0x00000017d52b0000
[    4.458664] CPU53: Booted secondary processor [410fd082]
[    4.541096] Detected PIPT I-cache on CPU54
[    4.541115] GICv3: CPU54: found redistributor 70102 region 3:0x000004006d280000
[    4.541161] CPU54: using LPI pending table @0x00000017d52e0000
[    4.541306] CPU54: Booted secondary processor [410fd082]
[    4.623738] Detected PIPT I-cache on CPU55
[    4.623757] GICv3: CPU55: found redistributor 70103 region 3:0x000004006d2c0000
[    4.623803] CPU55: using LPI pending table @0x00000017d5320000
[    4.623947] CPU55: Booted secondary processor [410fd082]
[    4.706381] Detected PIPT I-cache on CPU56
[    4.706403] GICv3: CPU56: found redistributor 70200 region 3:0x000004006d300000
[    4.706451] CPU56: using LPI pending table @0x00000017d5340000
[    4.706599] CPU56: Booted secondary processor [410fd082]
[    4.789020] Detected PIPT I-cache on CPU57
[    4.789040] GICv3: CPU57: found redistributor 70201 region 3:0x000004006d340000
[    4.789088] CPU57: using LPI pending table @0x00000017d5380000
[    4.789230] CPU57: Booted secondary processor [410fd082]
[    4.871663] Detected PIPT I-cache on CPU58
[    4.871682] GICv3: CPU58: found redistributor 70202 region 3:0x000004006d380000
[    4.871729] CPU58: using LPI pending table @0x00000017d53b0000
[    4.871874] CPU58: Booted secondary processor [410fd082]
[    4.954305] Detected PIPT I-cache on CPU59
[    4.954324] GICv3: CPU59: found redistributor 70203 region 3:0x000004006d3c0000
[    4.954370] CPU59: using LPI pending table @0x00000017d53e0000
[    4.954514] CPU59: Booted secondary processor [410fd082]
[    5.036947] Detected PIPT I-cache on CPU60
[    5.036970] GICv3: CPU60: found redistributor 70300 region 3:0x000004006d400000
[    5.037017] CPU60: using LPI pending table @0x00000017d4c10000
[    5.037164] CPU60: Booted secondary processor [410fd082]
[    5.119587] Detected PIPT I-cache on CPU61
[    5.119606] GICv3: CPU61: found redistributor 70301 region 3:0x000004006d440000
[    5.119652] CPU61: using LPI pending table @0x00000017d4c40000
[    5.119795] CPU61: Booted secondary processor [410fd082]
[    5.202229] Detected PIPT I-cache on CPU62
[    5.202249] GICv3: CPU62: found redistributor 70302 region 3:0x000004006d480000
[    5.202295] CPU62: using LPI pending table @0x00000017d4c70000
[    5.202441] CPU62: Booted secondary processor [410fd082]
[    5.284870] Detected PIPT I-cache on CPU63
[    5.284889] GICv3: CPU63: found redistributor 70303 region 3:0x000004006d4c0000
[    5.284936] CPU63: using LPI pending table @0x00000017d4cd0000
[    5.285078] CPU63: Booted secondary processor [410fd082]
[    5.285155] smp: Brought up 1 node, 64 CPUs
[    5.285297] SMP: Total of 64 processors activated.
[    5.285301] CPU features: detected feature: GIC system register CPU interface
[    5.285304] CPU features: detected feature: 32-bit EL0 Support
[    5.285541] CPU: All CPU(s) started at EL2
[    5.286196] devtmpfs: initialized
[    5.287679] SMBIOS 3.0.0 present.
[    5.287686] DMI: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI 16.12 Release 03/16/2017
[    5.287921] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    5.288154] futex hash table entries: 16384 (order: 9, 2097152 bytes)
[    5.288748] pinctrl core: initialized pinctrl subsystem
[    5.289166] NET: Registered protocol family 16
[    5.290546] cpuidle: using governor menu
[    5.290736] irq: no irq domain found for /interrupt-controller@a0080000/intc_smmu_pcie !
[    5.290901] arm-smmu-v3 a0040000.smmu_pcie: option hisilicon,broken-prefetch-cmd
[    5.290908] arm-smmu-v3 a0040000.smmu_pcie: ias 44-bit, oas 44-bit (features 0x00000f0d)
[    5.294474] irq: no irq domain found for /interrupt-controller@c0080000/intc_smmu_dsa !
[    5.294574] arm-smmu-v3 c0040000.smmu_dsa: option hisilicon,broken-prefetch-cmd
[    5.294589] arm-smmu-v3 c0040000.smmu_dsa: ias 44-bit, oas 44-bit (features 0x00000f0d)
[    5.298006] vdso: 2 pages (1 code @ ffff000008927000, 1 data @ ffff000008e25000)
[    5.298048] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    5.298984] DMA: preallocated 256 KiB pool for atomic allocations
[    5.299157] Serial: AMBA PL011 UART driver
[    5.299631] irq: no irq domain found for /interrupt-controller@60080000/uart_intc !
[    5.299710] irq: no irq domain found for /interrupt-controller@a0080000/intc_usb !
[    5.299850] iommu: Adding device a7030000.ohci to group 0
[    5.299893] irq: no irq domain found for /interrupt-controller@a0080000/intc_usb !
[    5.299960] iommu: Adding device a7020000.ehci to group 1
[    5.301608] irq: no irq domain found for /interrupt-controller@c0080000/intc_dsaf0 !
[    5.301731] iommu: Adding device c5000000.dsa to group 2
[    5.301885] irq: no irq domain found for /interrupt-controller@c0080000/intc-roce !
[    5.307388] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    5.308320] ACPI: Interpreter disabled.
[    5.308562] vgaarb: loaded
[    5.308650] SCSI subsystem initialized
[    5.308813] libata version 3.00 loaded.
[    5.308914] usbcore: registered new interface driver usbfs
[    5.308934] usbcore: registered new interface driver hub
[    5.309117] usbcore: registered new device driver usb
[    5.309229] pps_core: LinuxPPS API ver. 1 registered
[    5.309231] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    5.309239] PTP clock support registered
[    5.309323] Registered efivars operations
[    5.309503] Advanced Linux Sound Architecture Driver Initialized.
[    5.310075] clocksource: Switched to clocksource arch_sys_counter
[    5.310221] VFS: Disk quotas dquot_6.6.0
[    5.310250] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    5.310353] pnp: PnP ACPI: disabled
[    5.313377] NET: Registered protocol family 2
[    5.314108] TCP established hash table entries: 262144 (order: 9, 2097152 bytes)
[    5.314882] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    5.315277] TCP: Hash tables configured (established 262144 bind 65536)
[    5.315550] UDP hash table entries: 16384 (order: 7, 524288 bytes)
[    5.315787] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes)
[    5.316176] NET: Registered protocol family 1
[    5.316505] RPC: Registered named UNIX socket transport module.
[    5.316507] RPC: Registered udp transport module.
[    5.316510] RPC: Registered tcp transport module.
[    5.316511] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    5.316516] PCI: CLS 0 bytes, default 128
[    5.318900] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    5.319160] kvm [1]: 8-bit VMID
[    5.319163] kvm [1]: IDMAP page: 90e000
[    5.319166] kvm [1]: HYP VA range: 800000000000:ffffffffffff
[    5.320161] kvm [1]: Hyp mode initialized successfully
[    5.320274] kvm [1]: vgic-v2@fe020000
[    5.320366] kvm [1]: GIC system register CPU interface enabled
[    5.321105] kvm [1]: vgic interrupt IRQ1
[    5.321113] kvm [1]: virtual timer IRQ4
[    5.324516] audit: initializing netlink subsys (disabled)
[    5.324576] audit: type=2000 audit(3.644:1): state=initialized audit_enabled=0 res=1
[    5.324955] workingset: timestamp_bits=44 max_order=23 bucket_order=0
[    5.327563] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    5.327875] NFS: Registering the id_resolver key type
[    5.327888] Key type id_resolver registered
[    5.327891] Key type id_legacy registered
[    5.327897] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    5.327956] 9p: Installing v9fs 9p2000 file system support
[    5.329207] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    5.329212] io scheduler noop registered
[    5.329275] io scheduler cfq registered (default)
[    5.329278] io scheduler mq-deadline registered
[    5.331779] OF: PCI: host bridge /soc/pcie@a00a0000 ranges:
[    5.331789] OF: PCI:   MEM 0xa8800000..0xaffeffff -> 0xa8800000
[    5.331794] OF: PCI:    IO 0xafff0000..0xafffffff -> 0x00000000
[    5.331832] hisi-pcie-almost-ecam a8000000.pcie: ECAM at [mem 0xa8000000-0xa87fffff] for [bus 80-87]
[    5.331887] hisi-pcie-almost-ecam a8000000.pcie: PCI host bridge to bus 0000:80
[    5.331891] pci_bus 0000:80: root bus resource [bus 80-87]
[    5.331894] pci_bus 0000:80: root bus resource [mem 0xa8800000-0xaffeffff]
[    5.331897] pci_bus 0000:80: root bus resource [io  0x0000-0xffff]
[    5.331916] pci 0000:80:00.0: [19e5:1610] type 01 class 0x060400
[    5.331929] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.331935] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.331939] pci 0000:80:00.0: reg 0x10: [mem 0x00000000-0x0000ffff]
[    5.331942] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.331948] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.331953] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.331959] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.331990] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x52 may corrupt adjacent RW1C bits
[    5.332014] pci 0000:80:00.0: supports D1 D2
[    5.332017] pci 0000:80:00.0: PME# supported from D0 D1 D3hot
[    5.332021] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x44 may corrupt adjacent RW1C bits
[    5.332233] iommu: Adding device 0000:80:00.0 to group 3
[    5.332259] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x3e may corrupt adjacent RW1C bits
[    5.332265] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x3e may corrupt adjacent RW1C bits
[    5.332305] pci_bus 0000:81: busn_res: can not insert [bus 81-ff] under [bus 80-87] (conflicts with (null) [bus 80-87])
[    5.332322] pci 0000:81:00.0: [8086:10fb] type 00 class 0x020000
[    5.332339] pci 0000:81:00.0: reg 0x10: [mem 0xa9000000-0xa93fffff 64bit pref]
[    5.332345] pci 0000:81:00.0: reg 0x18: [io  0x0000-0x001f]
[    5.332360] pci 0000:81:00.0: reg 0x20: [mem 0xa9804000-0xa9807fff 64bit pref]
[    5.332367] pci 0000:81:00.0: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[    5.332433] pci 0000:81:00.0: PME# supported from D0 D3hot
[    5.332468] pci 0000:81:00.0: reg 0x184: [mem 0xa9704000-0xa9707fff 64bit pref]
[    5.332471] pci 0000:81:00.0: VF(n) BAR0 space: [mem 0xa9704000-0xa9803fff 64bit pref] (contains BAR0 for 64 VFs)
[    5.332488] pci 0000:81:00.0: reg 0x190: [mem 0xa9604000-0xa9607fff 64bit pref]
[    5.332490] pci 0000:81:00.0: VF(n) BAR3 space: [mem 0xa9604000-0xa9703fff 64bit pref] (contains BAR3 for 64 VFs)
[    5.332844] iommu: Adding device 0000:81:00.0 to group 4
[    5.332875] pci 0000:81:00.1: [8086:10fb] type 00 class 0x020000
[    5.332892] pci 0000:81:00.1: reg 0x10: [mem 0xa8c00000-0xa8ffffff 64bit pref]
[    5.332898] pci 0000:81:00.1: reg 0x18: [io  0x0000-0x001f]
[    5.332914] pci 0000:81:00.1: reg 0x20: [mem 0xa9600000-0xa9603fff 64bit pref]
[    5.332920] pci 0000:81:00.1: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[    5.332984] pci 0000:81:00.1: PME# supported from D0 D3hot
[    5.333012] pci 0000:81:00.1: reg 0x184: [mem 0xa9500000-0xa9503fff 64bit pref]
[    5.333015] pci 0000:81:00.1: VF(n) BAR0 space: [mem 0xa9500000-0xa95fffff 64bit pref] (contains BAR0 for 64 VFs)
[    5.333031] pci 0000:81:00.1: reg 0x190: [mem 0xa9400000-0xa9403fff 64bit pref]
[    5.333033] pci 0000:81:00.1: VF(n) BAR3 space: [mem 0xa9400000-0xa94fffff 64bit pref] (contains BAR3 for 64 VFs)
[    5.333322] iommu: Adding device 0000:81:00.1 to group 5
[    5.333344] pci_bus 0000:81: busn_res: [bus 81-ff] end is updated to 81
[    5.333365] pci 0000:80:00.0: BAR 14: assigned [mem 0xa8800000-0xa8ffffff]
[    5.333369] pci 0000:80:00.0: BAR 15: assigned [mem 0xa9000000-0xa9dfffff 64bit pref]
[    5.333372] pci 0000:80:00.0: BAR 0: assigned [mem 0xa9e00000-0xa9e0ffff]
[    5.333376] pci 0000:80:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    5.333381] pci 0000:81:00.0: BAR 0: assigned [mem 0xa9000000-0xa93fffff 64bit pref]
[    5.333390] pci 0000:81:00.0: BAR 6: assigned [mem 0xa8800000-0xa8bfffff pref]
[    5.333393] pci 0000:81:00.1: BAR 0: assigned [mem 0xa9400000-0xa97fffff 64bit pref]
[    5.333401] pci 0000:81:00.1: BAR 6: assigned [mem 0xa8c00000-0xa8ffffff pref]
[    5.333404] pci 0000:81:00.0: BAR 4: assigned [mem 0xa9800000-0xa9803fff 64bit pref]
[    5.333413] pci 0000:81:00.0: BAR 7: assigned [mem 0xa9804000-0xa9903fff 64bit pref]
[    5.333418] pci 0000:81:00.0: BAR 10: assigned [mem 0xa9904000-0xa9a03fff 64bit pref]
[    5.333423] pci 0000:81:00.1: BAR 4: assigned [mem 0xa9a04000-0xa9a07fff 64bit pref]
[    5.333432] pci 0000:81:00.1: BAR 7: assigned [mem 0xa9a08000-0xa9b07fff 64bit pref]
[    5.333437] pci 0000:81:00.1: BAR 10: assigned [mem 0xa9b08000-0xa9c07fff 64bit pref]
[    5.333442] pci 0000:81:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    5.333447] pci 0000:81:00.1: BAR 2: assigned [io  0x1020-0x103f]
[    5.333454] pci 0000:80:00.0: PCI bridge to [bus 81]
[    5.333458] pci 0000:80:00.0:   bridge window [io  0x1000-0x1fff]
[    5.333463] pci 0000:80:00.0:   bridge window [mem 0xa8800000-0xa8ffffff]
[    5.333466] pci 0000:80:00.0:   bridge window [mem 0xa9000000-0xa9dfffff 64bit pref]
[    5.333483] pcieport 0000:80:00.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    5.334485] pcieport 0000:80:00.0: Signaling PME with IRQ 7
[    5.334527] pcieport 0000:80:00.0: AER enabled with IRQ 7
[    5.334648] ipmi message handler version 39.2
[    5.334654] ipmi device interface
[    5.334694] IPMI System Interface driver.
[    5.334721] ipmi_si: probing via SMBIOS
[    5.334724] ipmi_si: SMBIOS: io 0xe4 regsize 1 spacing 1 irq 0
[    5.334728] ipmi_si: Adding SMBIOS-specified bt state machine
[    5.334732] ipmi_si: Trying SMBIOS-specified bt state machine at i/o address 0xe4, slave address 0x0, irq 0
[    5.334747] ipmi_si ipmi_si.0: Interface detection failed
[    5.335888] xenfs: not registering filesystem on non-xen platform
[    5.337169] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    5.337673] SuperH (H)SCI(F) driver initialized
[    5.337773] msm_serial: driver initialized
[    5.354990] loop: module loaded
[    5.355308] hisi_sas: driver version v1.6
[    5.356635] libphy: Fixed MDIO Bus: probed
[    5.357109] tun: Universal TUN/TAP device driver, 1.6
[    5.357111] tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
[    5.357535] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    5.357537] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    5.357592] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[    5.357594] igb: Copyright (c) 2007-2014 Intel Corporation.
[    5.357645] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
[    5.357647] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    5.357689] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.0.0-k
[    5.357691] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    5.357751] ixgbe 0000:81:00.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    5.358611] ixgbe 0000:81:00.0: enabling device (0000 -> 0002)
[    5.603275] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
[    5.603403] ixgbe 0000:81:00.0: PCI Express bandwidth of 32GT/s available
[    5.603407] ixgbe 0000:81:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    5.603492] ixgbe 0000:81:00.0: MAC: 2, PHY: 15, SFP+: 11, PBA No: FFFFFF-0FF
[    5.603495] ixgbe 0000:81:00.0: 68:a8:28:17:97:07
[    5.608297] ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection
[    5.608331] ixgbe 0000:81:00.1: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    5.609152] ixgbe 0000:81:00.1: enabling device (0000 -> 0002)
[    6.787178] ixgbe 0000:81:00.1: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
[    6.787307] ixgbe 0000:81:00.1: PCI Express bandwidth of 32GT/s available
[    6.787310] ixgbe 0000:81:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    6.787393] ixgbe 0000:81:00.1: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[    6.787395] ixgbe 0000:81:00.1: 68:a8:28:17:97:08
[    6.792053] ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection
[    6.792099] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k
[    6.792102] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[    6.792166] sky2: driver version 1.30
[    6.792615] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    6.792618] ehci-pci: EHCI PCI platform driver
[    6.792633] ehci-platform: EHCI generic platform driver
[    6.792663] ehci-platform a7020000.ehci: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    6.793574] ehci-platform a7020000.ehci: EHCI Host Controller
[    6.793604] ehci-platform a7020000.ehci: new USB bus registered, assigned bus number 1
[    6.793700] ehci-platform a7020000.ehci: irq 136, io mem 0xa7020000
[    6.806061] ehci-platform a7020000.ehci: USB 2.0 started, EHCI 1.00
[    6.806295] hub 1-0:1.0: USB hub found
[    6.806306] hub 1-0:1.0: 2 ports detected
[    6.806453] ehci-exynos: EHCI EXYNOS driver
[    6.806494] ehci-msm: Qualcomm On-Chip EHCI Host Controller
[    6.806519] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    6.806529] ohci-pci: OHCI PCI platform driver
[    6.806553] ohci-platform: OHCI generic platform driver
[    6.806573] ohci-platform a7030000.ohci: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    6.807473] ohci-platform a7030000.ohci: Generic Platform OHCI controller
[    6.807483] ohci-platform a7030000.ohci: new USB bus registered, assigned bus number 2
[    6.807546] ohci-platform a7030000.ohci: irq 137, io mem 0xa7030000
[    6.870326] hub 2-0:1.0: USB hub found
[    6.870338] hub 2-0:1.0: 2 ports detected
[    6.870465] ohci-exynos: OHCI EXYNOS driver
[    6.870633] usbcore: registered new interface driver usb-storage
[    6.997025] rtc-efi rtc-efi: rtc core: registered rtc-efi as rtc0
[    6.997205] i2c /dev entries driver
[    6.997868] sdhci: Secure Digital Host Controller Interface driver
[    6.997871] sdhci: Copyright(c) Pierre Ossman
[    6.997951] Synopsys Designware Multimedia Card Interface Driver
[    6.998132] sdhci-pltfm: SDHCI platform and OF driver helper
[    6.999163] ledtrig-cpu: registered to indicate activity on CPUs
[    6.999499] usbcore: registered new interface driver usbhid
[    6.999502] usbhid: USB HID core driver
[    7.000144] NET: Registered protocol family 17
[    7.000181] 9pnet: Installing 9P2000 support
[    7.000210] Key type dns_resolver registered
[    7.000460] registered taskstats version 1
[    7.000689] 602b0000.uart: ttyAMA0 at MMIO 0x602b0000 (irq = 138, base_baud = 0) is a SBSA
[    7.146062] usb 1-1: new high-speed USB device number 2 using ehci-platform
[    7.294762] hub 1-1:1.0: USB hub found
[    7.294856] hub 1-1:1.0: 4 ports detected
[    7.414056] usb 1-2: new high-speed USB device number 3 using ehci-platform
[    7.562759] hub 1-2:1.0: USB hub found
[    7.562855] hub 1-2:1.0: 4 ports detected
[    7.850053] usb 1-2.1: new full-speed USB device number 4 using ehci-platform
[    7.962516] input: Keyboard/Mouse KVM 1.1.0 as /devices/platform/soc/a7020000.ehci/usb1/1-2/1-2.1/1-2.1:1.0/0003:12D1:0003.0001/input/input0
[    8.022252] hid-generic 0003:12D1:0003.0001: input: USB HID v1.10 Keyboard [Keyboard/Mouse KVM 1.1.0] on usb-a7020000.ehci-2.1/input0
[    8.023043] input: Keyboard/Mouse KVM 1.1.0 as /devices/platform/soc/a7020000.ehci/usb1/1-2/1-2.1/1-2.1:1.1/0003:12D1:0003.0002/input/input1
[    8.023080] hid-generic 0003:12D1:0003.0002: input: USB HID v1.10 Mouse [Keyboard/Mouse KVM 1.1.0] on usb-a7020000.ehci-2.1/input1
[   10.972582] console [ttyAMA0] enabled
[   11.039405] rtc-efi rtc-efi: setting system clock to 2017-03-27 12:17:52 UTC (1490617072)
[   11.338770] ixgbe 0000:81:00.0: registered PHC device on eth0
[   11.515447] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
[   11.651534] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[   11.744809] ixgbe 0000:81:00.1: registered PHC device on eth1
[   11.878052] Sending DHCP requests ....
[   38.422250] random: fast init done
[   39.978047] .
[   52.514266] random: crng init done
[   63.554048] ., OK
[   64.572007] IP-Config: Got DHCP answer from 172.18.45.1, my address is 172.18.45.129
[   64.579746] IP-Config: Complete:
[   64.582974]      device=eth0, hwaddr=68:a8:28:17:97:07, ipaddr=172.18.45.129, mask=255.255.255.0, gw=172.18.45.1
[   64.593140]      host=172.18.45.129, domain=, nis-domain=(none)
[   64.599054]      bootserver=0.0.0.0, rootserver=172.18.45.166, rootpath=     nameserver0=8.8.8.8, nameserver1=4.4.4.4
[   64.610619] ixgbe 0000:81:00.1: removed PHC on eth1
[   65.018144] ALSA device list:
[   65.021102]   No soundcards found.
[   65.029925] VFS: Mounted root (nfs filesystem) on device 0:14.
[   65.036024] devtmpfs: mounted
[   65.039696] Freeing unused kernel memory: 1088K
[   65.322553] init: plymouth-upstart-bridge main process (1942) terminated with status 1
[   65.330532] init: plymouth-upstart-bridge main process ended, respawning
[   65.345855] init: plymouth-upstart-bridge main process (1992) terminated with status 1
[   65.354059] init: plymouth-upstart-bridge main process ended, respawning
[   65.544750] init: ureadahead main process (1945) terminated with status 5
[   73.154611] systemd-udevd[2193]: starting version 208
[   73.431168] ixgbe 0000:81:00.1 eth26: renamed from eth1
[   73.482158] systemd-udevd[2590]: renamed network interface eth1 to eth26
[   73.714196] Bluetooth: Core ver 2.22
[   73.714265] NET: Registered protocol family 31
[   73.714266] Bluetooth: HCI device and connection manager initialized
[   73.714279] Bluetooth: HCI socket layer initialized
[   73.714284] Bluetooth: L2CAP socket layer initialized
[   73.714299] Bluetooth: SCO socket layer initialized
[   74.373037] init: failsafe main process (2723) killed by TERM signal
[   74.485356] NET: Registered protocol family 10
[   74.488119] Segment Routing with IPv6
[   76.520945] ip_tables: (C) 2000-2006 Netfilter Core Team
[   76.624565] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   76.935734] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   77.915416] init: tty1 main process (3310) killed by TERM signal
[   78.048409] init: plymouth-upstart-bridge main process ended, respawning
[  142.404919] ixgbe 0000:81:00.0 eth0: SR-IOV enabled with 1 VFs
[  142.405042] ixgbe 0000:81:00.0: removed PHC on eth0
[  142.614933] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[  142.635890] ixgbe 0000:81:00.0: registered PHC device on eth0
[  142.808352] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
[  142.851412] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[  142.851906] iommu: Adding device 0000:81:10.0 to group 6
[  142.852063] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[  142.852836] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  142.852962] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
[  142.853833] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  142.863956] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  142.863960] ixgbevf 0000:81:10.0: Assigning random MAC address
[  142.865689] ixgbevf 0000:81:10.0: da:9f:f8:1e:57:3a
[  142.865692] ixgbevf 0000:81:10.0: MAC: 1
[  142.865693] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  142.939145] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[  152.902894] nfs: server 172.18.45.166 not responding, still trying
[  188.980933] nfs: server 172.18.45.166 not responding, still trying
[  188.981298] nfs: server 172.18.45.166 OK
[  188.981593] nfs: server 172.18.45.166 OK
[  221.755626] VFIO - User Level meta-driver version: 0.3
[    0.000000] Booting Linux on physical CPU 0x10000
[    0.000000] Linux version 4.11.0-rc1-g2802757-dirty (shameer@shameer-ubuntu) (gcc version 4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 - Linaro GCC 4.9-2014.09) ) #154 SMP PREEMPT Mon Mar 27 12:24:59 BST 2017
[    0.000000] Boot CPU: AArch64 Processor [410fd082]
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: EFI v2.60 by EDK II
[    0.000000] efi:  SMBIOS=0x3f050000  SMBIOS 3.0=0x39b00000  ACPI=0x39bd0000  ACPI 2.0=0x39bd0014  MEMATTR=0x3c1b6018 
[    0.000000] cma: Reserved 16 MiB at 0x000000003e000000
[    0.000000] NUMA: Adding memblock [0x0 - 0x3fffffff] on node 0
[    0.000000] OF: NUMA: parsing numa-distance-map-v1
[    0.000000] NUMA: Warning: invalid memblk node 4 [mem 0x1040000000-0x17fbffffff]
[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x00000017fbffffff]
[    0.000000] NUMA: Adding memblock [0x0 - 0x257ff] on node 0
[    0.000000] NUMA: Adding memblock [0x25800 - 0x3187ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x31880000 - 0x3188ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x31890000 - 0x318fafff] on node 0
[    0.000000] NUMA: Adding memblock [0x318fb000 - 0x318fefff] on node 0
[    0.000000] NUMA: Adding memblock [0x318ff000 - 0x3999ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x399a0000 - 0x3a05ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x3a060000 - 0x3f04ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x3f050000 - 0x3f07ffff] on node 0
[    0.000000] NUMA: Adding memblock [0x3f080000 - 0x3fbfffff] on node 0
[    0.000000] NUMA: Adding memblock [0x1040000000 - 0x17fbffffff] on node 0
[    0.000000] NUMA: Initmem setup node 0 [mem 0x00000000-0x17fbffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x17fbfc4d00-0x17fbfc67ff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000017fbffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000000024fff]
[    0.000000]   node   0: [mem 0x0000000000026000-0x000000003187ffff]
[    0.000000]   node   0: [mem 0x0000000031880000-0x000000003188ffff]
[    0.000000]   node   0: [mem 0x0000000031890000-0x00000000318fafff]
[    0.000000]   node   0: [mem 0x00000000318fb000-0x00000000318fefff]
[    0.000000]   node   0: [mem 0x00000000318ff000-0x000000003999ffff]
[    0.000000]   node   0: [mem 0x00000000399a0000-0x000000003a05ffff]
[    0.000000]   node   0: [mem 0x000000003a060000-0x000000003f04ffff]
[    0.000000]   node   0: [mem 0x000000003f050000-0x000000003f07ffff]
[    0.000000]   node   0: [mem 0x000000003f080000-0x000000003fbfffff]
[    0.000000]   node   0: [mem 0x0000001040000000-0x00000017fbffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000017fbffffff]
[    0.000000] On node 0 totalpages: 8371199
[    0.000000]   DMA zone: 4080 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 261119 pages, LIFO batch:31
[    0.000000]   Normal zone: 126720 pages used for memmap
[    0.000000]   Normal zone: 8110080 pages, LIFO batch:31
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] percpu: Embedded 23 pages/cpu @ffff8017db640000 s56960 r8192 d29056 u94208
[    0.000000] pcpu-alloc: s56960 r8192 d29056 u94208 alloc=23*4096
[    0.000000] pcpu-alloc: [0] 00 [0] 01 [0] 02 [0] 03 [0] 04 [0] 05 [0] 06 [0] 07 
[    0.000000] pcpu-alloc: [0] 08 [0] 09 [0] 10 [0] 11 [0] 12 [0] 13 [0] 14 [0] 15 
[    0.000000] pcpu-alloc: [0] 16 [0] 17 [0] 18 [0] 19 [0] 20 [0] 21 [0] 22 [0] 23 
[    0.000000] pcpu-alloc: [0] 24 [0] 25 [0] 26 [0] 27 [0] 28 [0] 29 [0] 30 [0] 31 
[    0.000000] pcpu-alloc: [0] 32 [0] 33 [0] 34 [0] 35 [0] 36 [0] 37 [0] 38 [0] 39 
[    0.000000] pcpu-alloc: [0] 40 [0] 41 [0] 42 [0] 43 [0] 44 [0] 45 [0] 46 [0] 47 
[    0.000000] pcpu-alloc: [0] 48 [0] 49 [0] 50 [0] 51 [0] 52 [0] 53 [0] 54 [0] 55 
[    0.000000] pcpu-alloc: [0] 56 [0] 57 [0] 58 [0] 59 [0] 60 [0] 61 [0] 62 [0] 63 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 8240399
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/shameer/Image_smmu_4.11 rdinit=/init console=ttyAMA0,115200 earlycon=sbsa-uart,mmio,0x602b0000 pcie_aspm=off acpi=off noinitrd root=/dev/nfs rw nfsroot=172.18.45.166:/home/shameer/sherlok-files/ubuntu-sher/ubuntu,nfsvers=3 ip=dhcp::eth4:on:172.18.45.1:8.8.8.8 user_debug=1 nfsrootdebug pcie_acs_override=downstream
[    0.000000] PCIe ASPM is disabled
[    0.000000] log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] log_buf_len total cpu_extra contributions: 258048 bytes
[    0.000000] log_buf_len min size: 131072 bytes
[    0.000000] log_buf_len: 524288 bytes
[    0.000000] early log buf free: 125900(96%)
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] software IO TLB [mem 0x359a0000-0x399a0000] (64MB) mapped at [ffff8000359a0000-ffff80003999ffff]
[    0.000000] Memory: 32848756K/33484796K available (8828K kernel code, 964K rwdata, 3976K rodata, 1088K init, 398K bss, 619656K reserved, 16384K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000   (   128 MB)
[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000   (129022 GB)
[    0.000000]       .text : 0xffff000008080000 - 0xffff000008920000   (  8832 KB)
[    0.000000]     .rodata : 0xffff000008920000 - 0xffff000008d10000   (  4032 KB)
[    0.000000]       .init : 0xffff000008d10000 - 0xffff000008e20000   (  1088 KB)
[    0.000000]       .data : 0xffff000008e20000 - 0xffff000008f11200   (   965 KB)
[    0.000000]        .bss : 0xffff000008f11200 - 0xffff000008f74db8   (   399 KB)
[    0.000000]     fixed   : 0xffff7dfffe7fd000 - 0xffff7dfffec00000   (  4108 KB)
[    0.000000]     PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000   (    16 MB)
[    0.000000]     vmemmap : 0xffff7e0000000000 - 0xffff800000000000   (  2048 GB maximum)
[    0.000000]               0xffff7e0000000000 - 0xffff7e005ff00000   (  1535 MB actual)
[    0.000000]     memory  : 0xffff800000000000 - 0xffff8017fc000000   ( 98240 MB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=64, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] 	Build-time adjustment of leaf fanout to 64.
[    0.000000] NR_IRQS:64 nr_irqs:64 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] ITS [mem 0x4c000000-0x4c03ffff]
[    0.000000] ITS@0x000000004c000000: allocated 524288 Devices @17da800000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000000004c000000: allocated 2048 Virtual CPUs @17db024000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000000004c000000: allocated 512 Interrupt Collections @17db021000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x6c000000-0x6c03ffff]
[    0.000000] ITS@0x000000006c000000: allocated 524288 Devices @17da400000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000000006c000000: allocated 2048 Virtual CPUs @17db028000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000000006c000000: allocated 512 Interrupt Collections @17db023000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0xc6000000-0xc603ffff]
[    0.000000] ITS@0x00000000c6000000: allocated 524288 Devices @17da000000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000000c6000000: allocated 2048 Virtual CPUs @17db02c000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000000c6000000: allocated 512 Interrupt Collections @17db060000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x8c6000000-0x8c603ffff]
[    0.000000] ITS@0x00000008c6000000: allocated 524288 Devices @17d9c00000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000008c6000000: allocated 2048 Virtual CPUs @17db064000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000008c6000000: allocated 512 Interrupt Collections @17db061000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x4004c000000-0x4004c03ffff]
[    0.000000] ITS@0x000004004c000000: allocated 524288 Devices @17d9800000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000004004c000000: allocated 2048 Virtual CPUs @17db068000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000004004c000000: allocated 512 Interrupt Collections @17db062000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x4006c000000-0x4006c03ffff]
[    0.000000] ITS@0x000004006c000000: allocated 524288 Devices @17d9400000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x000004006c000000: allocated 2048 Virtual CPUs @17db06c000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x000004006c000000: allocated 512 Interrupt Collections @17db0a0000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x400c6000000-0x400c603ffff]
[    0.000000] ITS@0x00000400c6000000: allocated 524288 Devices @17d9000000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000400c6000000: allocated 2048 Virtual CPUs @17db0a4000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000400c6000000: allocated 512 Interrupt Collections @17db0a1000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS [mem 0x408c6000000-0x408c603ffff]
[    0.000000] ITS@0x00000408c6000000: allocated 524288 Devices @17d8c00000 (flat, esz 8, psz 16K, shr 1)
[    0.000000] ITS@0x00000408c6000000: allocated 2048 Virtual CPUs @17db0a8000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] ITS@0x00000408c6000000: allocated 512 Interrupt Collections @17db0a2000 (flat, esz 8, psz 4K, shr 1)
[    0.000000] GIC: using LPI property table @0x00000017db0d0000
[    0.000000] ITS: Allocated 1792 chunks for LPIs
[    0.000000] GICv3: CPU0: found redistributor 10000 region 0:0x000000004d100000
[    0.000000] CPU0: using LPI pending table @0x00000017db0e0000
[    0.000000] arm_arch_timer: Architected cp15 timer(s) running at 50.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
[    0.000001] sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
[    0.000107] Console: colour dummy device 80x25
[    0.000139] Calibrating delay loop (skipped), value calculated using timer frequency.. 100.00 BogoMIPS (lpj=200000)
[    0.000143] pid_max: default: 65536 minimum: 512
[    0.000182] Security Framework initialized
[    0.003835] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
[    0.013291] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.017172] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.017230] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.043989] ASID allocator initialised with 65536 entries
[    0.064017] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@4c000000 domain created
[    0.064021] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@6c000000 domain created
[    0.064024] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@c6000000 domain created
[    0.064027] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@8,c6000000 domain created
[    0.064030] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@400,4c000000 domain created
[    0.064033] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@400,6c000000 domain created
[    0.064036] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@400,c6000000 domain created
[    0.064039] PCI/MSI: /interrupt-controller@4d000000/interrupt-controller@408,c6000000 domain created
[    0.064065] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@4c000000 domain created
[    0.064068] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@6c000000 domain created
[    0.064071] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@c6000000 domain created
[    0.064074] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@8,c6000000 domain created
[    0.064077] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@400,4c000000 domain created
[    0.064080] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@400,6c000000 domain created
[    0.064083] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@400,c6000000 domain created
[    0.064086] Platform MSI: /interrupt-controller@4d000000/interrupt-controller@408,c6000000 domain created
[    0.064272] Remapping and enabling EFI services.
[    0.064309]   EFI remap 0x00000000399a0000 => 0000000020000000
[    0.064331]   EFI remap 0x00000000399f0000 => 0000000020050000
[    0.064340]   EFI remap 0x0000000039a40000 => 00000000200a0000
[    0.064370]   EFI remap 0x0000000039af0000 => 00000000200b0000
[    0.064392]   EFI remap 0x0000000039b70000 => 0000000020130000
[    0.064413]   EFI remap 0x0000000039be0000 => 0000000020180000
[    0.064439]   EFI remap 0x0000000039c30000 => 00000000201d0000
[    0.064460]   EFI remap 0x0000000039c80000 => 0000000020220000
[    0.064482]   EFI remap 0x0000000039cd0000 => 0000000020270000
[    0.064504]   EFI remap 0x0000000039d20000 => 00000000202c0000
[    0.064525]   EFI remap 0x0000000039d70000 => 0000000020310000
[    0.064546]   EFI remap 0x0000000039dc0000 => 0000000020360000
[    0.064567]   EFI remap 0x0000000039e10000 => 00000000203b0000
[    0.064597]   EFI remap 0x0000000039e60000 => 0000000020400000
[    0.064619]   EFI remap 0x0000000039ed0000 => 0000000020470000
[    0.064641]   EFI remap 0x0000000039f20000 => 00000000204c0000
[    0.064662]   EFI remap 0x0000000039f70000 => 0000000020510000
[    0.064684]   EFI remap 0x0000000039fc0000 => 0000000020560000
[    0.064705]   EFI remap 0x000000003a010000 => 00000000205b0000
[    0.064723]   EFI remap 0x000000003f050000 => 0000000020600000
[    0.064732]   EFI remap 0x0000000078000000 => 0000000020630000
[    0.064738]   EFI remap 0x00000000a4000000 => 0000000020800000
[    0.064750]   EFI remap 0x00000000a6000000 => 0000000021800000
[    0.064759]   EFI remap 0x00000000d00e0000 => 0000000021810000
[    0.080009] smp: Bringing up secondary CPUs ...
[    0.162611] Detected PIPT I-cache on CPU1
[    0.162623] GICv3: CPU1: found redistributor 10001 region 0:0x000000004d140000
[    0.162644] CPU1: using LPI pending table @0x00000017d5850000
[    0.162790] CPU1: Booted secondary processor [410fd082]
[    0.245231] Detected PIPT I-cache on CPU2
[    0.245237] GICv3: CPU2: found redistributor 10002 region 0:0x000000004d180000
[    0.245260] CPU2: using LPI pending table @0x00000017d5880000
[    0.245397] CPU2: Booted secondary processor [410fd082]
[    0.327852] Detected PIPT I-cache on CPU3
[    0.327857] GICv3: CPU3: found redistributor 10003 region 0:0x000000004d1c0000
[    0.327880] CPU3: using LPI pending table @0x00000017d58c0000
[    0.328018] CPU3: Booted secondary processor [410fd082]
[    0.410477] Detected PIPT I-cache on CPU4
[    0.410487] GICv3: CPU4: found redistributor 10100 region 0:0x000000004d200000
[    0.410510] CPU4: using LPI pending table @0x00000017d58f0000
[    0.410651] CPU4: Booted secondary processor [410fd082]
[    0.493098] Detected PIPT I-cache on CPU5
[    0.493104] GICv3: CPU5: found redistributor 10101 region 0:0x000000004d240000
[    0.493127] CPU5: using LPI pending table @0x00000017d5920000
[    0.493265] CPU5: Booted secondary processor [410fd082]
[    0.575722] Detected PIPT I-cache on CPU6
[    0.575728] GICv3: CPU6: found redistributor 10102 region 0:0x000000004d280000
[    0.575752] CPU6: using LPI pending table @0x00000017d5950000
[    0.575889] CPU6: Booted secondary processor [410fd082]
[    0.658346] Detected PIPT I-cache on CPU7
[    0.658353] GICv3: CPU7: found redistributor 10103 region 0:0x000000004d2c0000
[    0.658374] CPU7: using LPI pending table @0x00000017d5980000
[    0.658512] CPU7: Booted secondary processor [410fd082]
[    0.740973] Detected PIPT I-cache on CPU8
[    0.740983] GICv3: CPU8: found redistributor 10200 region 0:0x000000004d300000
[    0.741006] CPU8: using LPI pending table @0x00000017d59b0000
[    0.741147] CPU8: Booted secondary processor [410fd082]
[    0.823595] Detected PIPT I-cache on CPU9
[    0.823602] GICv3: CPU9: found redistributor 10201 region 0:0x000000004d340000
[    0.823624] CPU9: using LPI pending table @0x00000017d59f0000
[    0.823762] CPU9: Booted secondary processor [410fd082]
[    0.906220] Detected PIPT I-cache on CPU10
[    0.906227] GICv3: CPU10: found redistributor 10202 region 0:0x000000004d380000
[    0.906249] CPU10: using LPI pending table @0x00000017d5a20000
[    0.906387] CPU10: Booted secondary processor [410fd082]
[    0.988845] Detected PIPT I-cache on CPU11
[    0.988851] GICv3: CPU11: found redistributor 10203 region 0:0x000000004d3c0000
[    0.988874] CPU11: using LPI pending table @0x00000017d5a70000
[    0.989013] CPU11: Booted secondary processor [410fd082]
[    1.071473] Detected PIPT I-cache on CPU12
[    1.071484] GICv3: CPU12: found redistributor 10300 region 0:0x000000004d400000
[    1.071507] CPU12: using LPI pending table @0x00000017d5aa0000
[    1.071649] CPU12: Booted secondary processor [410fd082]
[    1.154094] Detected PIPT I-cache on CPU13
[    1.154101] GICv3: CPU13: found redistributor 10301 region 0:0x000000004d440000
[    1.154124] CPU13: using LPI pending table @0x00000017d5ad0000
[    1.154262] CPU13: Booted secondary processor [410fd082]
[    1.236720] Detected PIPT I-cache on CPU14
[    1.236727] GICv3: CPU14: found redistributor 10302 region 0:0x000000004d480000
[    1.236749] CPU14: using LPI pending table @0x00000017d5b00000
[    1.236889] CPU14: Booted secondary processor [410fd082]
[    1.319344] Detected PIPT I-cache on CPU15
[    1.319351] GICv3: CPU15: found redistributor 10303 region 0:0x000000004d4c0000
[    1.319375] CPU15: using LPI pending table @0x00000017d5b40000
[    1.319513] CPU15: Booted secondary processor [410fd082]
[    1.401620] Detected PIPT I-cache on CPU16
[    1.401629] GICv3: CPU16: found redistributor 30000 region 1:0x000000006d100000
[    1.401646] CPU16: using LPI pending table @0x00000017d5b70000
[    1.401750] CPU16: Booted secondary processor [410fd082]
[    1.484249] Detected PIPT I-cache on CPU17
[    1.484256] GICv3: CPU17: found redistributor 30001 region 1:0x000000006d140000
[    1.484273] CPU17: using LPI pending table @0x00000017d5b90000
[    1.484379] CPU17: Booted secondary processor [410fd082]
[    1.566878] Detected PIPT I-cache on CPU18
[    1.566886] GICv3: CPU18: found redistributor 30002 region 1:0x000000006d180000
[    1.566903] CPU18: using LPI pending table @0x00000017d5bd0000
[    1.567010] CPU18: Booted secondary processor [410fd082]
[    1.649507] Detected PIPT I-cache on CPU19
[    1.649514] GICv3: CPU19: found redistributor 30003 region 1:0x000000006d1c0000
[    1.649532] CPU19: using LPI pending table @0x00000017d5400000
[    1.649637] CPU19: Booted secondary processor [410fd082]
[    1.732137] Detected PIPT I-cache on CPU20
[    1.732145] GICv3: CPU20: found redistributor 30100 region 1:0x000000006d200000
[    1.732163] CPU20: using LPI pending table @0x00000017d5430000
[    1.732269] CPU20: Booted secondary processor [410fd082]
[    1.814765] Detected PIPT I-cache on CPU21
[    1.814773] GICv3: CPU21: found redistributor 30101 region 1:0x000000006d240000
[    1.814789] CPU21: using LPI pending table @0x00000017d5460000
[    1.814894] CPU21: Booted secondary processor [410fd082]
[    1.897395] Detected PIPT I-cache on CPU22
[    1.897403] GICv3: CPU22: found redistributor 30102 region 1:0x000000006d280000
[    1.897419] CPU22: using LPI pending table @0x00000017d5490000
[    1.897523] CPU22: Booted secondary processor [410fd082]
[    1.980024] Detected PIPT I-cache on CPU23
[    1.980031] GICv3: CPU23: found redistributor 30103 region 1:0x000000006d2c0000
[    1.980048] CPU23: using LPI pending table @0x00000017d54c0000
[    1.980153] CPU23: Booted secondary processor [410fd082]
[    2.062655] Detected PIPT I-cache on CPU24
[    2.062664] GICv3: CPU24: found redistributor 30200 region 1:0x000000006d300000
[    2.062682] CPU24: using LPI pending table @0x00000017d54f0000
[    2.062789] CPU24: Booted secondary processor [410fd082]
[    2.145282] Detected PIPT I-cache on CPU25
[    2.145290] GICv3: CPU25: found redistributor 30201 region 1:0x000000006d340000
[    2.145306] CPU25: using LPI pending table @0x00000017d5530000
[    2.145412] CPU25: Booted secondary processor [410fd082]
[    2.227912] Detected PIPT I-cache on CPU26
[    2.227920] GICv3: CPU26: found redistributor 30202 region 1:0x000000006d380000
[    2.227938] CPU26: using LPI pending table @0x00000017d5560000
[    2.228043] CPU26: Booted secondary processor [410fd082]
[    2.310540] Detected PIPT I-cache on CPU27
[    2.310548] GICv3: CPU27: found redistributor 30203 region 1:0x000000006d3c0000
[    2.310565] CPU27: using LPI pending table @0x00000017d5590000
[    2.310671] CPU27: Booted secondary processor [410fd082]
[    2.393171] Detected PIPT I-cache on CPU28
[    2.393180] GICv3: CPU28: found redistributor 30300 region 1:0x000000006d400000
[    2.393199] CPU28: using LPI pending table @0x00000017d55c0000
[    2.393304] CPU28: Booted secondary processor [410fd082]
[    2.475798] Detected PIPT I-cache on CPU29
[    2.475806] GICv3: CPU29: found redistributor 30301 region 1:0x000000006d440000
[    2.475823] CPU29: using LPI pending table @0x00000017d55f0000
[    2.475928] CPU29: Booted secondary processor [410fd082]
[    2.558427] Detected PIPT I-cache on CPU30
[    2.558435] GICv3: CPU30: found redistributor 30302 region 1:0x000000006d480000
[    2.558453] CPU30: using LPI pending table @0x00000017d5630000
[    2.558557] CPU30: Booted secondary processor [410fd082]
[    2.641057] Detected PIPT I-cache on CPU31
[    2.641065] GICv3: CPU31: found redistributor 30303 region 1:0x000000006d4c0000
[    2.641082] CPU31: using LPI pending table @0x00000017d5660000
[    2.641187] CPU31: Booted secondary processor [410fd082]
[    2.723368] Detected PIPT I-cache on CPU32
[    2.723404] GICv3: CPU32: found redistributor 50000 region 2:0x000004004d100000
[    2.723454] CPU32: using LPI pending table @0x00000017d5690000
[    2.723603] CPU32: Booted secondary processor [410fd082]
[    2.805990] Detected PIPT I-cache on CPU33
[    2.806007] GICv3: CPU33: found redistributor 50001 region 2:0x000004004d140000
[    2.806050] CPU33: using LPI pending table @0x00000017d56c0000
[    2.806154] CPU33: Booted secondary processor [410fd082]
[    2.888629] Detected PIPT I-cache on CPU34
[    2.888646] GICv3: CPU34: found redistributor 50002 region 2:0x000004004d180000
[    2.888688] CPU34: using LPI pending table @0x00000017d56f0000
[    2.888794] CPU34: Booted secondary processor [410fd082]
[    2.971267] Detected PIPT I-cache on CPU35
[    2.971284] GICv3: CPU35: found redistributor 50003 region 2:0x000004004d1c0000
[    2.971325] CPU35: using LPI pending table @0x00000017d5720000
[    2.971430] CPU35: Booted secondary processor [410fd082]
[    3.053909] Detected PIPT I-cache on CPU36
[    3.053926] GICv3: CPU36: found redistributor 50100 region 2:0x000004004d200000
[    3.053968] CPU36: using LPI pending table @0x00000017d5760000
[    3.054075] CPU36: Booted secondary processor [410fd082]
[    3.136549] Detected PIPT I-cache on CPU37
[    3.136565] GICv3: CPU37: found redistributor 50101 region 2:0x000004004d240000
[    3.136608] CPU37: using LPI pending table @0x00000017d5790000
[    3.136712] CPU37: Booted secondary processor [410fd082]
[    3.219189] Detected PIPT I-cache on CPU38
[    3.219205] GICv3: CPU38: found redistributor 50102 region 2:0x000004004d280000
[    3.219248] CPU38: using LPI pending table @0x00000017d57c0000
[    3.219353] CPU38: Booted secondary processor [410fd082]
[    3.301832] Detected PIPT I-cache on CPU39
[    3.301848] GICv3: CPU39: found redistributor 50103 region 2:0x000004004d2c0000
[    3.301889] CPU39: using LPI pending table @0x00000017d57f0000
[    3.301994] CPU39: Booted secondary processor [410fd082]
[    3.384474] Detected PIPT I-cache on CPU40
[    3.384492] GICv3: CPU40: found redistributor 50200 region 2:0x000004004d300000
[    3.384533] CPU40: using LPI pending table @0x00000017d5020000
[    3.384641] CPU40: Booted secondary processor [410fd082]
[    3.467112] Detected PIPT I-cache on CPU41
[    3.467129] GICv3: CPU41: found redistributor 50201 region 2:0x000004004d340000
[    3.467171] CPU41: using LPI pending table @0x00000017d5050000
[    3.467276] CPU41: Booted secondary processor [410fd082]
[    3.549753] Detected PIPT I-cache on CPU42
[    3.549769] GICv3: CPU42: found redistributor 50202 region 2:0x000004004d380000
[    3.549813] CPU42: using LPI pending table @0x00000017d5080000
[    3.549920] CPU42: Booted secondary processor [410fd082]
[    3.632393] Detected PIPT I-cache on CPU43
[    3.632409] GICv3: CPU43: found redistributor 50203 region 2:0x000004004d3c0000
[    3.632454] CPU43: using LPI pending table @0x00000017d50c0000
[    3.632560] CPU43: Booted secondary processor [410fd082]
[    3.715033] Detected PIPT I-cache on CPU44
[    3.715051] GICv3: CPU44: found redistributor 50300 region 2:0x000004004d400000
[    3.715095] CPU44: using LPI pending table @0x00000017d50e0000
[    3.715203] CPU44: Booted secondary processor [410fd082]
[    3.797676] Detected PIPT I-cache on CPU45
[    3.797693] GICv3: CPU45: found redistributor 50301 region 2:0x000004004d440000
[    3.797735] CPU45: using LPI pending table @0x00000017d5120000
[    3.797840] CPU45: Booted secondary processor [410fd082]
[    3.880317] Detected PIPT I-cache on CPU46
[    3.880334] GICv3: CPU46: found redistributor 50302 region 2:0x000004004d480000
[    3.880377] CPU46: using LPI pending table @0x00000017d5150000
[    3.880484] CPU46: Booted secondary processor [410fd082]
[    3.962956] Detected PIPT I-cache on CPU47
[    3.962973] GICv3: CPU47: found redistributor 50303 region 2:0x000004004d4c0000
[    3.963016] CPU47: using LPI pending table @0x00000017d5180000
[    3.963122] CPU47: Booted secondary processor [410fd082]
[    4.045257] Detected PIPT I-cache on CPU48
[    4.045280] GICv3: CPU48: found redistributor 70000 region 3:0x000004006d100000
[    4.045326] CPU48: using LPI pending table @0x00000017d51b0000
[    4.045474] CPU48: Booted secondary processor [410fd082]
[    4.127897] Detected PIPT I-cache on CPU49
[    4.127916] GICv3: CPU49: found redistributor 70001 region 3:0x000004006d140000
[    4.127962] CPU49: using LPI pending table @0x00000017d51e0000
[    4.128107] CPU49: Booted secondary processor [410fd082]
[    4.210539] Detected PIPT I-cache on CPU50
[    4.210557] GICv3: CPU50: found redistributor 70002 region 3:0x000004006d180000
[    4.210603] CPU50: using LPI pending table @0x00000017d5210000
[    4.210749] CPU50: Booted secondary processor [410fd082]
[    4.293181] Detected PIPT I-cache on CPU51
[    4.293200] GICv3: CPU51: found redistributor 70003 region 3:0x000004006d1c0000
[    4.293246] CPU51: using LPI pending table @0x00000017d5250000
[    4.293389] CPU51: Booted secondary processor [410fd082]
[    4.375826] Detected PIPT I-cache on CPU52
[    4.375848] GICv3: CPU52: found redistributor 70100 region 3:0x000004006d200000
[    4.375895] CPU52: using LPI pending table @0x00000017d5280000
[    4.376043] CPU52: Booted secondary processor [410fd082]
[    4.458466] Detected PIPT I-cache on CPU53
[    4.458485] GICv3: CPU53: found redistributor 70101 region 3:0x000004006d240000
[    4.458533] CPU53: using LPI pending table @0x00000017d52b0000
[    4.458677] CPU53: Booted secondary processor [410fd082]
[    4.541110] Detected PIPT I-cache on CPU54
[    4.541129] GICv3: CPU54: found redistributor 70102 region 3:0x000004006d280000
[    4.541176] CPU54: using LPI pending table @0x00000017d52e0000
[    4.541322] CPU54: Booted secondary processor [410fd082]
[    4.623752] Detected PIPT I-cache on CPU55
[    4.623771] GICv3: CPU55: found redistributor 70103 region 3:0x000004006d2c0000
[    4.623817] CPU55: using LPI pending table @0x00000017d5320000
[    4.623961] CPU55: Booted secondary processor [410fd082]
[    4.706397] Detected PIPT I-cache on CPU56
[    4.706419] GICv3: CPU56: found redistributor 70200 region 3:0x000004006d300000
[    4.706468] CPU56: using LPI pending table @0x00000017d5340000
[    4.706617] CPU56: Booted secondary processor [410fd082]
[    4.789037] Detected PIPT I-cache on CPU57
[    4.789056] GICv3: CPU57: found redistributor 70201 region 3:0x000004006d340000
[    4.789104] CPU57: using LPI pending table @0x00000017d5380000
[    4.789249] CPU57: Booted secondary processor [410fd082]
[    4.871681] Detected PIPT I-cache on CPU58
[    4.871700] GICv3: CPU58: found redistributor 70202 region 3:0x000004006d380000
[    4.871746] CPU58: using LPI pending table @0x00000017d53b0000
[    4.871892] CPU58: Booted secondary processor [410fd082]
[    4.954321] Detected PIPT I-cache on CPU59
[    4.954340] GICv3: CPU59: found redistributor 70203 region 3:0x000004006d3c0000
[    4.954390] CPU59: using LPI pending table @0x00000017d53e0000
[    4.954532] CPU59: Booted secondary processor [410fd082]
[    5.036966] Detected PIPT I-cache on CPU60
[    5.036989] GICv3: CPU60: found redistributor 70300 region 3:0x000004006d400000
[    5.037035] CPU60: using LPI pending table @0x00000017d4c10000
[    5.037183] CPU60: Booted secondary processor [410fd082]
[    5.119607] Detected PIPT I-cache on CPU61
[    5.119626] GICv3: CPU61: found redistributor 70301 region 3:0x000004006d440000
[    5.119674] CPU61: using LPI pending table @0x00000017d4c40000
[    5.119818] CPU61: Booted secondary processor [410fd082]
[    5.202251] Detected PIPT I-cache on CPU62
[    5.202270] GICv3: CPU62: found redistributor 70302 region 3:0x000004006d480000
[    5.202319] CPU62: using LPI pending table @0x00000017d4c70000
[    5.202465] CPU62: Booted secondary processor [410fd082]
[    5.284892] Detected PIPT I-cache on CPU63
[    5.284911] GICv3: CPU63: found redistributor 70303 region 3:0x000004006d4c0000
[    5.284958] CPU63: using LPI pending table @0x00000017d4cd0000
[    5.285100] CPU63: Booted secondary processor [410fd082]
[    5.285177] smp: Brought up 1 node, 64 CPUs
[    5.285322] SMP: Total of 64 processors activated.
[    5.285326] CPU features: detected feature: GIC system register CPU interface
[    5.285329] CPU features: detected feature: 32-bit EL0 Support
[    5.285561] CPU: All CPU(s) started at EL2
[    5.286204] devtmpfs: initialized
[    5.287662] SMBIOS 3.0.0 present.
[    5.287668] DMI: Huawei Taishan 2280 /D05, BIOS Hisilicon D05 UEFI 16.12 Release 03/16/2017
[    5.287902] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    5.288140] futex hash table entries: 16384 (order: 9, 2097152 bytes)
[    5.288719] pinctrl core: initialized pinctrl subsystem
[    5.289150] NET: Registered protocol family 16
[    5.290547] cpuidle: using governor menu
[    5.290742] irq: no irq domain found for /interrupt-controller@a0080000/intc_smmu_pcie !
[    5.290904] arm-smmu-v3 a0040000.smmu_pcie: option hisilicon,broken-prefetch-cmd
[    5.290911] arm-smmu-v3 a0040000.smmu_pcie: ias 44-bit, oas 44-bit (features 0x00000f0d)
[    5.294485] irq: no irq domain found for /interrupt-controller@c0080000/intc_smmu_dsa !
[    5.294581] arm-smmu-v3 c0040000.smmu_dsa: option hisilicon,broken-prefetch-cmd
[    5.294585] arm-smmu-v3 c0040000.smmu_dsa: ias 44-bit, oas 44-bit (features 0x00000f0d)
[    5.298023] vdso: 2 pages (1 code @ ffff000008927000, 1 data @ ffff000008e25000)
[    5.298064] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    5.299021] DMA: preallocated 256 KiB pool for atomic allocations
[    5.299230] Serial: AMBA PL011 UART driver
[    5.299700] irq: no irq domain found for /interrupt-controller@60080000/uart_intc !
[    5.299774] irq: no irq domain found for /interrupt-controller@a0080000/intc_usb !
[    5.299822] irq: no irq domain found for /interrupt-controller@a0080000/intc_usb !
[    5.301494] irq: no irq domain found for /interrupt-controller@c0080000/intc_dsaf0 !
[    5.301660] irq: no irq domain found for /interrupt-controller@c0080000/intc-roce !
[    5.307230] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    5.308232] ACPI: Interpreter disabled.
[    5.308486] vgaarb: loaded
[    5.308575] SCSI subsystem initialized
[    5.308770] libata version 3.00 loaded.
[    5.308867] usbcore: registered new interface driver usbfs
[    5.308888] usbcore: registered new interface driver hub
[    5.309071] usbcore: registered new device driver usb
[    5.309193] pps_core: LinuxPPS API ver. 1 registered
[    5.309195] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    5.309205] PTP clock support registered
[    5.309296] Registered efivars operations
[    5.309483] Advanced Linux Sound Architecture Driver Initialized.
[    5.310080] clocksource: Switched to clocksource arch_sys_counter
[    5.310222] VFS: Disk quotas dquot_6.6.0
[    5.310250] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    5.310353] pnp: PnP ACPI: disabled
[    5.313513] NET: Registered protocol family 2
[    5.314224] TCP established hash table entries: 262144 (order: 9, 2097152 bytes)
[    5.315000] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    5.315419] TCP: Hash tables configured (established 262144 bind 65536)
[    5.315699] UDP hash table entries: 16384 (order: 7, 524288 bytes)
[    5.315925] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes)
[    5.316362] NET: Registered protocol family 1
[    5.316735] RPC: Registered named UNIX socket transport module.
[    5.316738] RPC: Registered udp transport module.
[    5.316740] RPC: Registered tcp transport module.
[    5.316742] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    5.316746] PCI: CLS 0 bytes, default 128
[    5.319057] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    5.319313] kvm [1]: 8-bit VMID
[    5.319316] kvm [1]: IDMAP page: 90e000
[    5.319319] kvm [1]: HYP VA range: 800000000000:ffffffffffff
[    5.320299] kvm [1]: Hyp mode initialized successfully
[    5.320413] kvm [1]: vgic-v2@fe020000
[    5.320504] kvm [1]: GIC system register CPU interface enabled
[    5.321247] kvm [1]: vgic interrupt IRQ1
[    5.321256] kvm [1]: virtual timer IRQ4
[    5.324764] audit: initializing netlink subsys (disabled)
[    5.324833] audit: type=2000 audit(3.644:1): state=initialized audit_enabled=0 res=1
[    5.325243] workingset: timestamp_bits=44 max_order=23 bucket_order=0
[    5.328073] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    5.328424] NFS: Registering the id_resolver key type
[    5.328438] Key type id_resolver registered
[    5.328440] Key type id_legacy registered
[    5.328447] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    5.328511] 9p: Installing v9fs 9p2000 file system support
[    5.329843] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    5.329848] io scheduler noop registered
[    5.329917] io scheduler cfq registered (default)
[    5.329920] io scheduler mq-deadline registered
[    5.332537] OF: PCI: host bridge /soc/pcie@a00a0000 ranges:
[    5.332546] OF: PCI:   MEM 0xa8800000..0xaffeffff -> 0xa8800000
[    5.332551] OF: PCI:    IO 0xafff0000..0xafffffff -> 0x00000000
[    5.332590] hisi-pcie-almost-ecam a8000000.pcie: ECAM at [mem 0xa8000000-0xa87fffff] for [bus 80-87]
[    5.332642] hisi-pcie-almost-ecam a8000000.pcie: PCI host bridge to bus 0000:80
[    5.332646] pci_bus 0000:80: root bus resource [bus 80-87]
[    5.332649] pci_bus 0000:80: root bus resource [mem 0xa8800000-0xaffeffff]
[    5.332652] pci_bus 0000:80: root bus resource [io  0x0000-0xffff]
[    5.332668] pci 0000:80:00.0: [19e5:1610] type 01 class 0x060400
[    5.332681] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.332688] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.332693] pci 0000:80:00.0: reg 0x10: [mem 0x00000000-0x0000ffff]
[    5.332696] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.332702] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.332706] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.332712] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    5.332730] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x52 may corrupt adjacent RW1C bits
[    5.332754] pci 0000:80:00.0: supports D1 D2
[    5.332757] pci 0000:80:00.0: PME# supported from D0 D1 D3hot
[    5.332760] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x44 may corrupt adjacent RW1C bits
[    5.332844] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x3e may corrupt adjacent RW1C bits
[    5.332849] pci_bus 0000:80: 2-byte config write to 0000:80:00.0 offset 0x3e may corrupt adjacent RW1C bits
[    5.332886] pci_bus 0000:81: busn_res: can not insert [bus 81-ff] under [bus 80-87] (conflicts with (null) [bus 80-87])
[    5.332903] pci 0000:81:00.0: [8086:10fb] type 00 class 0x020000
[    5.332919] pci 0000:81:00.0: reg 0x10: [mem 0xa9000000-0xa93fffff 64bit pref]
[    5.332925] pci 0000:81:00.0: reg 0x18: [io  0x0000-0x001f]
[    5.332940] pci 0000:81:00.0: reg 0x20: [mem 0xa9804000-0xa9807fff 64bit pref]
[    5.332946] pci 0000:81:00.0: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[    5.333005] pci 0000:81:00.0: PME# supported from D0 D3hot
[    5.333040] pci 0000:81:00.0: reg 0x184: [mem 0xa9704000-0xa9707fff 64bit pref]
[    5.333043] pci 0000:81:00.0: VF(n) BAR0 space: [mem 0xa9704000-0xa9803fff 64bit pref] (contains BAR0 for 64 VFs)
[    5.333058] pci 0000:81:00.0: reg 0x190: [mem 0xa9604000-0xa9607fff 64bit pref]
[    5.333061] pci 0000:81:00.0: VF(n) BAR3 space: [mem 0xa9604000-0xa9703fff 64bit pref] (contains BAR3 for 64 VFs)
[    5.333341] pci 0000:81:00.1: [8086:10fb] type 00 class 0x020000
[    5.333357] pci 0000:81:00.1: reg 0x10: [mem 0xa8c00000-0xa8ffffff 64bit pref]
[    5.333364] pci 0000:81:00.1: reg 0x18: [io  0x0000-0x001f]
[    5.333378] pci 0000:81:00.1: reg 0x20: [mem 0xa9600000-0xa9603fff 64bit pref]
[    5.333385] pci 0000:81:00.1: reg 0x30: [mem 0xffc00000-0xffffffff pref]
[    5.333443] pci 0000:81:00.1: PME# supported from D0 D3hot
[    5.333471] pci 0000:81:00.1: reg 0x184: [mem 0xa9500000-0xa9503fff 64bit pref]
[    5.333474] pci 0000:81:00.1: VF(n) BAR0 space: [mem 0xa9500000-0xa95fffff 64bit pref] (contains BAR0 for 64 VFs)
[    5.333489] pci 0000:81:00.1: reg 0x190: [mem 0xa9400000-0xa9403fff 64bit pref]
[    5.333492] pci 0000:81:00.1: VF(n) BAR3 space: [mem 0xa9400000-0xa94fffff 64bit pref] (contains BAR3 for 64 VFs)
[    5.333750] pci_bus 0000:81: busn_res: [bus 81-ff] end is updated to 81
[    5.333771] pci 0000:80:00.0: BAR 14: assigned [mem 0xa8800000-0xa8ffffff]
[    5.333775] pci 0000:80:00.0: BAR 15: assigned [mem 0xa9000000-0xa9dfffff 64bit pref]
[    5.333777] pci 0000:80:00.0: BAR 0: assigned [mem 0xa9e00000-0xa9e0ffff]
[    5.333782] pci 0000:80:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    5.333787] pci 0000:81:00.0: BAR 0: assigned [mem 0xa9000000-0xa93fffff 64bit pref]
[    5.333796] pci 0000:81:00.0: BAR 6: assigned [mem 0xa8800000-0xa8bfffff pref]
[    5.333799] pci 0000:81:00.1: BAR 0: assigned [mem 0xa9400000-0xa97fffff 64bit pref]
[    5.333808] pci 0000:81:00.1: BAR 6: assigned [mem 0xa8c00000-0xa8ffffff pref]
[    5.333811] pci 0000:81:00.0: BAR 4: assigned [mem 0xa9800000-0xa9803fff 64bit pref]
[    5.333820] pci 0000:81:00.0: BAR 7: assigned [mem 0xa9804000-0xa9903fff 64bit pref]
[    5.333825] pci 0000:81:00.0: BAR 10: assigned [mem 0xa9904000-0xa9a03fff 64bit pref]
[    5.333830] pci 0000:81:00.1: BAR 4: assigned [mem 0xa9a04000-0xa9a07fff 64bit pref]
[    5.333838] pci 0000:81:00.1: BAR 7: assigned [mem 0xa9a08000-0xa9b07fff 64bit pref]
[    5.333844] pci 0000:81:00.1: BAR 10: assigned [mem 0xa9b08000-0xa9c07fff 64bit pref]
[    5.333848] pci 0000:81:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    5.333853] pci 0000:81:00.1: BAR 2: assigned [io  0x1020-0x103f]
[    5.333860] pci 0000:80:00.0: PCI bridge to [bus 81]
[    5.333864] pci 0000:80:00.0:   bridge window [io  0x1000-0x1fff]
[    5.333869] pci 0000:80:00.0:   bridge window [mem 0xa8800000-0xa8ffffff]
[    5.333872] pci 0000:80:00.0:   bridge window [mem 0xa9000000-0xa9dfffff 64bit pref]
[    5.334042] iommu: Adding device 0000:80:00.0 to group 0
[    5.334053] pcieport 0000:80:00.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    5.335031] pcieport 0000:80:00.0: Signaling PME with IRQ 7
[    5.335077] pcieport 0000:80:00.0: AER enabled with IRQ 7
[    5.335200] ipmi message handler version 39.2
[    5.335206] ipmi device interface
[    5.335248] IPMI System Interface driver.
[    5.335276] ipmi_si: probing via SMBIOS
[    5.335278] ipmi_si: SMBIOS: io 0xe4 regsize 1 spacing 1 irq 0
[    5.335282] ipmi_si: Adding SMBIOS-specified bt state machine
[    5.335287] ipmi_si: Trying SMBIOS-specified bt state machine at i/o address 0xe4, slave address 0x0, irq 0
[    5.335303] ipmi_si ipmi_si.0: Interface detection failed
[    5.336401] xenfs: not registering filesystem on non-xen platform
[    5.337625] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    5.338157] SuperH (H)SCI(F) driver initialized
[    5.338254] msm_serial: driver initialized
[    5.355682] loop: module loaded
[    5.356033] hisi_sas: driver version v1.6
[    5.356792] libphy: Fixed MDIO Bus: probed
[    5.357045] tun: Universal TUN/TAP device driver, 1.6
[    5.357047] tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
[    5.357460] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    5.357463] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    5.357524] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[    5.357526] igb: Copyright (c) 2007-2014 Intel Corporation.
[    5.357586] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k
[    5.357588] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    5.357638] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.0.0-k
[    5.357640] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    5.357875] iommu: Adding device 0000:81:00.0 to group 1
[    5.357885] ixgbe 0000:81:00.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    5.358742] ixgbe 0000:81:00.0: enabling device (0000 -> 0002)
[    5.519236] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
[    5.519365] ixgbe 0000:81:00.0: PCI Express bandwidth of 32GT/s available
[    5.519368] ixgbe 0000:81:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    5.519452] ixgbe 0000:81:00.0: MAC: 2, PHY: 15, SFP+: 11, PBA No: FFFFFF-0FF
[    5.519455] ixgbe 0000:81:00.0: 68:a8:28:17:97:07
[    5.524366] ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection
[    5.524491] iommu: Adding device 0000:81:00.1 to group 2
[    5.524499] ixgbe 0000:81:00.1: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    5.525327] ixgbe 0000:81:00.1: enabling device (0000 -> 0002)
[    6.703188] ixgbe 0000:81:00.1: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63
[    6.703316] ixgbe 0000:81:00.1: PCI Express bandwidth of 32GT/s available
[    6.703319] ixgbe 0000:81:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[    6.703402] ixgbe 0000:81:00.1: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[    6.703405] ixgbe 0000:81:00.1: 68:a8:28:17:97:08
[    6.708048] ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection
[    6.708094] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k
[    6.708097] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[    6.708175] sky2: driver version 1.30
[    6.708616] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    6.708619] ehci-pci: EHCI PCI platform driver
[    6.708635] ehci-platform: EHCI generic platform driver
[    6.708802] iommu: Adding device a7020000.ehci to group 3
[    6.708810] ehci-platform a7020000.ehci: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    6.709700] ehci-platform a7020000.ehci: EHCI Host Controller
[    6.709729] ehci-platform a7020000.ehci: new USB bus registered, assigned bus number 1
[    6.709820] ehci-platform a7020000.ehci: irq 136, io mem 0xa7020000
[    6.722071] ehci-platform a7020000.ehci: USB 2.0 started, EHCI 1.00
[    6.722310] hub 1-0:1.0: USB hub found
[    6.722321] hub 1-0:1.0: 2 ports detected
[    6.722462] ehci-exynos: EHCI EXYNOS driver
[    6.722498] ehci-msm: Qualcomm On-Chip EHCI Host Controller
[    6.722523] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    6.722533] ohci-pci: OHCI PCI platform driver
[    6.722555] ohci-platform: OHCI generic platform driver
[    6.722653] iommu: Adding device a7030000.ohci to group 4
[    6.722661] ohci-platform a7030000.ohci: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[    6.723537] ohci-platform a7030000.ohci: Generic Platform OHCI controller
[    6.723546] ohci-platform a7030000.ohci: new USB bus registered, assigned bus number 2
[    6.723611] ohci-platform a7030000.ohci: irq 137, io mem 0xa7030000
[    6.786404] hub 2-0:1.0: USB hub found
[    6.786418] hub 2-0:1.0: 2 ports detected
[    6.786582] ohci-exynos: OHCI EXYNOS driver
[    6.786804] usbcore: registered new interface driver usb-storage
[    6.913217] rtc-efi rtc-efi: rtc core: registered rtc-efi as rtc0
[    6.913377] i2c /dev entries driver
[    6.913990] sdhci: Secure Digital Host Controller Interface driver
[    6.913992] sdhci: Copyright(c) Pierre Ossman
[    6.914073] Synopsys Designware Multimedia Card Interface Driver
[    6.914237] sdhci-pltfm: SDHCI platform and OF driver helper
[    6.915259] ledtrig-cpu: registered to indicate activity on CPUs
[    6.915544] usbcore: registered new interface driver usbhid
[    6.915547] usbhid: USB HID core driver
[    6.916117] NET: Registered protocol family 17
[    6.916146] 9pnet: Installing 9P2000 support
[    6.916170] Key type dns_resolver registered
[    6.916411] registered taskstats version 1
[    6.916613] 602b0000.uart: ttyAMA0 at MMIO 0x602b0000 (irq = 138, base_baud = 0) is a SBSA
[    7.058068] usb 1-1: new high-speed USB device number 2 using ehci-platform
[    7.058209] ehci-platform a7020000.ehci: fatal error
[    7.058212] ehci-platform a7020000.ehci: HC died; cleaning up
[    7.442068] usb 2-1: new full-speed USB device number 2 using ohci-platform
[    7.653099] usb 2-1: not running at top speed; connect to a high speed hub
[    7.659126] hub 2-1:1.0: USB hub found
[    7.661100] hub 2-1:1.0: 4 ports detected
[    7.850064] usb 2-2: new full-speed USB device number 3 using ohci-platform
[    8.061098] usb 2-2: not running at top speed; connect to a high speed hub
[    8.067126] hub 2-2:1.0: USB hub found
[    8.069099] hub 2-2:1.0: 4 ports detected
[    8.374063] usb 2-2.1: new full-speed USB device number 4 using ohci-platform
[    8.509436] input: Keyboard/Mouse KVM 1.1.0 as /devices/platform/soc/a7030000.ohci/usb2/2-2/2-2.1/2-2.1:1.0/0003:12D1:0003.0001/input/input0
[    8.566249] hid-generic 0003:12D1:0003.0001: input: USB HID v1.10 Keyboard [Keyboard/Mouse KVM 1.1.0] on usb-a7030000.ohci-2.1/input0
[    8.569382] input: Keyboard/Mouse KVM 1.1.0 as /devices/platform/soc/a7030000.ohci/usb2/2-2/2-2.1/2-2.1:1.1/0003:12D1:0003.0002/input/input1
[    8.569428] hid-generic 0003:12D1:0003.0002: input: USB HID v1.10 Mouse [Keyboard/Mouse KVM 1.1.0] on usb-a7030000.ohci-2.1/input1
[   10.914206] console [ttyAMA0] enabled
[   10.980942] rtc-efi rtc-efi: setting system clock to 2017-03-27 11:34:34 UTC (1490614474)
[   11.271062] ixgbe 0000:81:00.0: registered PHC device on eth0
[   11.447464] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
[   11.583544] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[   11.677525] ixgbe 0000:81:00.1: registered PHC device on eth1
[   11.810063] Sending DHCP requests .....
[   37.402083] random: fast init done
[   55.946395] random: crng init done
[   58.874060] ., OK
[   59.884016] IP-Config: Got DHCP answer from 172.18.45.1, my address is 172.18.45.129
[   59.891754] IP-Config: Complete:
[   59.894976]      device=eth0, hwaddr=68:a8:28:17:97:07, ipaddr=172.18.45.129, mask=255.255.255.0, gw=172.18.45.1
[   59.905147]      host=172.18.45.129, domain=, nis-domain=(none)
[   59.911059]      bootserver=0.0.0.0, rootserver=172.18.45.166, rootpath=     nameserver0=8.8.8.8, nameserver1=4.4.4.4
[   59.922614] ixgbe 0000:81:00.1: removed PHC on eth1
[   60.337680] ALSA device list:
[   60.340652]   No soundcards found.
[   60.349483] VFS: Mounted root (nfs filesystem) on device 0:14.
[   60.355689] devtmpfs: mounted
[   60.359404] Freeing unused kernel memory: 1088K
[   60.550258] init: plymouth-upstart-bridge main process (1942) terminated with status 1
[   60.558209] init: plymouth-upstart-bridge main process ended, respawning
[   60.721227] init: ureadahead main process (1945) terminated with status 5
[   67.644936] systemd-udevd[2171]: starting version 208
[   67.943594] ixgbe 0000:81:00.1 eth26: renamed from eth1
[   68.058186] systemd-udevd[2574]: renamed network interface eth1 to eth26
[   68.121180] Bluetooth: Core ver 2.22
[   68.121281] NET: Registered protocol family 31
[   68.121282] Bluetooth: HCI device and connection manager initialized
[   68.121293] Bluetooth: HCI socket layer initialized
[   68.121298] Bluetooth: L2CAP socket layer initialized
[   68.121312] Bluetooth: SCO socket layer initialized
[   68.319028] init: failsafe main process (2687) killed by TERM signal
[   68.753718] NET: Registered protocol family 10
[   68.756503] Segment Routing with IPv6
[   69.971495] ip_tables: (C) 2000-2006 Netfilter Core Team
[   70.078809] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   70.197725] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   71.194210] init: tty1 main process (3319) killed by TERM signal
[   71.415444] init: plymouth-upstart-bridge main process ended, respawning
[  144.741629] ixgbe 0000:81:00.0 eth0: SR-IOV enabled with 1 VFs
[  144.741799] ixgbe 0000:81:00.0: removed PHC on eth0
[  144.977338] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[  144.997161] ixgbe 0000:81:00.0: registered PHC device on eth0
[  145.169155] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
[  145.211717] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[  145.212351] iommu: Adding device 0000:81:10.0 to group 5
[  145.212367] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[  145.213261] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  145.213394] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
[  145.214272] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  145.224379] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  145.224384] ixgbevf 0000:81:10.0: Assigning random MAC address
[  145.225941] ixgbevf 0000:81:10.0: 1a:85:06:48:a7:19
[  145.225944] ixgbevf 0000:81:10.0: MAC: 1
[  145.225946] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  145.299961] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[  154.947742] nfs: server 172.18.45.166 not responding, still trying
[  191.025780] nfs: server 172.18.45.166 not responding, still trying
[  191.026122] nfs: server 172.18.45.166 OK
[  191.026317] nfs: server 172.18.45.166 OK
[  263.706402] VFIO - User Level meta-driver version: 0.3
[  269.757613] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
[  269.757617] specified DMA range outside IOMMU capability
[  269.757618] Failed to set up IOMMU for device 0000:81:10.0; retaining platform DMA ops

Just to add to that, the mask is set to 64 bit in the ixgebvf driver probe[1]

[  127.914001] ixgbe 0000:81:00.0 eth0: SR-IOV enabled with 1 VFs
[  127.914106] ixgbe 0000:81:00.0: removed PHC on eth0
[  128.125166] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4
[  128.143857] ixgbe 0000:81:00.0: registered PHC device on eth0
[  128.314754] ixgbe 0000:81:00.0 eth0: detected SFP+: 11
[  128.357878] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[  128.358416] iommu: Adding device 0000:81:10.0 to group 5
[  128.358443] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
[  128.359326] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  128.359333] Shameer: ixgbevf_probe, 64 bit			------------->mask set to 64 bit
[  128.359462] ixgbe 0000:81:00.0 eth0: VF Reset msg received from vf 0
[  128.360331] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  128.370470] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  128.370474] ixgbevf 0000:81:10.0: Assigning random MAC address
[  128.372172] ixgbevf 0000:81:10.0: ea:40:b9:e9:cb:04
[  128.372176] ixgbevf 0000:81:10.0: MAC: 1
[  128.372178] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  128.445551] ixgbe 0000:81:00.0 eth0: NIC Link is Up 1 Gbps, Flow Control: None
[  138.089869] nfs: server 172.18.45.166 not responding, still trying
[  174.697868] nfs: server 172.18.45.166 not responding, still trying
[  174.698359] nfs: server 172.18.45.166 OK
[  174.698582] nfs: server 172.18.45.166 OK
[  465.942259] VFIO - User Level meta-driver version: 0.3
 [  472.754074] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff
[  472.754075] specified DMA range outside IOMMU capability
[  472.754077] Failed to set up IOMMU for device 0000:81:10.0; retaining platform DMA ops

1. http://lxr.free-electrons.com/source/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c#L3996

Aha, but of course it's still the same struct device getting bound to VFIO later, so whatever mask the first driver set is still in there when we go through of_dma_configure() the second time (and the fact that we go through more than once being the new behaviour). So yes, this is a legitimate problem and we really do need to be robust against size overflow. I reckon the below tweak of your fix is probably the way to go; cleaning up the arch_setup_dma_ops() interface can happen later.

Robin.

-----8<-----

index 9933077df7b7..77d080bde52d 100644
@@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct

device_node *np)
ret = of_dma_get_range(np, &dma_addr, &paddr, &size); if (ret < 0) {
dma_addr = offset = 0;
- size = dev->coherent_dma_mask + 1;
+ size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
} else {
offset = PFN_DOWN(paddr - dma_addr);

----->8-----

On Mon, Mar 27, 2017 at 05:18:15PM +0100, Robin Murphy wrote:

[...]

Yes. On ACPI the additional issue is, given that we now call *_dma_configure multiple times, we end up adding the _same_ stream id to an existing fwspec ids array which triggers the SMMUv3 driver bug, because we try to configure the same id twice (it does not happen in DT since you have a check similar to the one I added below).

ACPI fix(es) below please let me know if it actually fixes the issues:

-- >8 --

index 36a9abf..e7b1940 100644
@@ -543,6 +543,14 @@ static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 	const struct iommu_ops *ops = NULL;
 	int ret = -ENODEV;
 	struct fwnode_handle *iort_fwnode;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
+
+	/*
+	 * If we already translated the fwspec there
+	 * is nothing left to do, return the iommu_ops.
+	 */
+	if (fwspec && fwspec->ops)
+		return fwspec->ops;
 
 	if (node) {
 		iort_fwnode = iort_get_fwnode(node);

index 823b005..2a513cc 100644
@@ -1376,18 +1376,20 @@ enum dev_dma_attr acpi_get_dma_attr(struct acpi_device *adev)
 int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
 {
 	const struct iommu_ops *iommu;
+	u64 size;
 
 	iort_set_dma_mask(dev);
 
 	iommu = iort_iommu_configure(dev);
 	if (IS_ERR(iommu))
 		return PTR_ERR(iommu);
+
+	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	/*
 	 * Assume dma valid range starts at 0 and covers the whole
 	 * coherent_dma_mask.
 	 */
-	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
-			   attr == DEV_DMA_COHERENT);
+	arch_setup_dma_ops(dev, 0, size, iommu, attr == DEV_DMA_COHERENT);
 
 	return 0;
 }

Hi,

ok, i will add this fix separately and also the acpi fix that lorenzo has suggested in patch #8 in to the series after testing confirmation.

Regards,
Sricharan

[...]

I can confirm that the patches fixes the issues reported here . Both DT and ACPI works now.

Cheers,
Shameer

Hi,

Thanks for the testing. Will repost with the fixes.

Regards,
Sricharan

"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

This series calls the dma ops configuration for the devices at a generic place so that it works for all busses. The dma_configure_ops for a device is now called during the device_attach callback just before the probe of the bus/driver is called. Similarly dma_deconfigure is called during device/driver_detach path.

pci_bus_add_devices (platform/amba)(_device_create/driver_register)

       |                         |
pci_bus_add_device     (device_add/driver_register)
       |                         |
device_attach           device_initial_probe
       |                         |
__device_attach_driver    __device_attach_driver
       |
driver_probe_device
       |
really_probe
       |

dma_configure

Similarly on the device/driver_unregister path __device_release_driver is called which inturn calls dma_deconfigure.

Rebased the series against mainline 4.11-rc5. Applies and builds cleanly against mainline and linux-next, iommu-next.

* Tested with platform and pci devices for probe deferral and reprobe on arm64 based platform.

Previous post of this series [8].

Please note that, i have kept the tested/acked tags intact from V8 because V9/10 were for more fixes that was added, so the original tags that was given for the functional testing remains the same.

[V10]
* Rebased on top of 4.11-rc5.

* Fixed coherent_dma_mask 64bit overflow issue [8] for OF. The fix for OF was added as a separate patch#6, since the issue is true even without probe deferral, but gets reproduced with the probe deferral series. Added Lorenzo's ACPI fix for coherent_dma_mask overflow and the fix for dma_configure getting called more than once for the same device.

* Also fixed an build issue caught by kbuild robot for m68k arch. The issue was dma_(de)configure was not getting defined for !CONFIG_HAS_DMA, so fixed that as well.

[V9]
* Rebased on top of 4.11-rc1.

* Merged Robin's fixes for legacy binding issue, pci devices with no iommu-map property and deferencing of_iommu_table after init.

[V8]
* Picked up all the acks and tested tags from Marek and Hanjun for DT and ACPI patches respectively, since no functional changes was done.

* Addressed Minor comments Sinan and Bjorn.

* Added Robin's fix for fixing the deferencing NULL for of_iommu_table after init in patch #2.

* Rebased it on top of linux-next

[V7]
* Updated the subject and commit log for patch #6 as per comments from Lorenzo. No functional changes.

[V6]
* Fixed a bug in dma_configure function pointed out by Robin.
* Reordered the patches as per comments from Robin and Lorenzo.
* Added Tags.

[V5]
* Reworked the pci configuration code hanging outside and pushed it to dma_configure as in PATCH#5,6,7. Also added a couple of patches that Lorenzo provided for correcting the Probe deferring mechanism in case of ACPI devices from here [5].

[V4]
* Took the reworked patches [2] from Robin's branch and rebased on top of Lorenzo's ACPI IORT ARM support series [3].

* Added the patches for moving the dma ops configuration of acpi based devices to probe time as well.
[V3]
* Removed the patch to split dma_masks/dma_ops configuration separately based on review comments that both masks and ops are required only during the device probe time.

* Reworked the series based on Generic DT bindings series.

* Added call to iommu's remove_device in the cleanup path for arm and arm64.

* Removed the notifier trick in arm64 to handle early device registration.

* Added reset of dma_ops in cleanup path for arm based on comments.

* Fixed the pci_iommu_configure path and tested with PCI device as well.

* Fixed a bug to return the correct iommu_ops from patch 7 [4] in last post.

* Fixed few other cosmetic comments.

[V2]
* Updated the Initial post to call dma_configure/deconfigure from generic code

* Added iommu add_device callback from of_iommu_configure path

[V1]
* Initial post from Laurent Pinchart [1]

[1] http://lists.linuxfoundation.org/pipermail/iommu/2015-May/013016.html
[2] http://www.linux-arm.org/git?p=linux-rm.git;a=shortlog;h=refs/heads/iommu/defer
[3] https://lkml.org/lkml/2016/11/21/141
[4] https://www.mail-archive.com/iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx/msg13940.html
[5] git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/linux.git iommu/probe-deferral
[6] http://www.spinics.net/lists/linux-pci/msg57992.html
[7] https://www.spinics.net/lists/arm-kernel/msg556209.html
[8] http://patchwork.ozlabs.org/patch/743898/

Laurent Pinchart (3):
of: dma: Move range size workaround to of_dma_get_range()
of: dma: Make of_dma_deconfigure() public
iommu: of: Handle IOMMU lookup failure with deferred probing or error

Lorenzo Pieralisi (2):
ACPI/IORT: Add function to check SMMUs drivers presence ACPI/IORT: Remove linker section for IORT entries probing

Robin Murphy (3):
iommu/of: Refactor of_iommu_configure() for error handling iommu/of: Prepare for deferred IOMMU configuration iommu/arm-smmu: Clean up early-probing workarounds

Sricharan R (4):
of: device: Fix overflow of coherent_dma_mask
of/acpi: Configure dma operations at probe time for platform/amba/pci
bus devices
drivers: acpi: Handle IOMMU lookup failure with deferred probing or
error
arm64: dma-mapping: Remove the notifier trick to handle early setting
of dma_ops

 arch/arm64/mm/dma-mapping.c       | 142 +++++---------------------------------
 drivers/acpi/arm64/iort.c         |  48 ++++++++++++-
 drivers/acpi/glue.c               |   5 --
 drivers/acpi/scan.c               |  11 ++-
 drivers/base/dd.c                 |   9 +++
 drivers/base/dma-mapping.c        |  41 +++++++++++
 drivers/iommu/arm-smmu-v3.c       |  46 +-----------
 drivers/iommu/arm-smmu.c          | 110 +++++++++++++----------------
 drivers/iommu/of_iommu.c          | 126 ++++++++++++++++++++++++---------
 drivers/of/address.c              |  20 +++++-
 drivers/of/device.c               |  36 +++++-----
 drivers/of/platform.c             |  10 +--
 drivers/pci/probe.c               |  28 --------
 include/acpi/acpi_bus.h           |   2 +-
 include/asm-generic/vmlinux.lds.h |   1 -
 include/linux/acpi.h              |   7 +-
 include/linux/acpi_iort.h         |   3 -
 include/linux/dma-mapping.h       |  12 ++++
 include/linux/of_device.h         |  10 ++-
 19 files changed, 329 insertions(+), 338 deletions(-)

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

From: Robin Murphy <[email protected]>

In preparation for some upcoming cleverness, rework the control flow in of_iommu_configure() to minimise duplication and improve the propogation of errors. It's also as good a time as any to switch over from the now-just-a-compatibility-wrapper of_iommu_get_ops() to using the generic IOMMU instance interface directly.

---
 drivers/iommu/of_iommu.c | 83 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 30 deletions(-)

index 2683e9f..8f4e599 100644
@@ -96,6 +96,28 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static const struct iommu_ops
+*of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
+{
+	const struct iommu_ops *ops;
+	struct fwnode_handle *fwnode = &iommu_spec->np->fwnode;
+	int err;
+
+	ops = iommu_ops_from_fwnode(fwnode);
+	if (!ops || !ops->of_xlate)
+		return NULL;
+
+	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
+	if (err)
+		return ERR_PTR(err);
+
+	err = ops->of_xlate(dev, iommu_spec);
+	if (err)
+		return ERR_PTR(err);
+
+	return ops;
+}
+
 static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 {
 	struct of_phandle_args *iommu_spec = data;

@@ -105,10 +127,11 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 }
 
 static const struct iommu_ops
-*of_pci_iommu_configure(struct pci_dev *pdev, struct device_node *bridge_np)
+*of_pci_iommu_init(struct pci_dev *pdev, struct device_node *bridge_np)
 {
 	const struct iommu_ops *ops;
 	struct of_phandle_args iommu_spec;
+	int err;
 
 	/*
 	 * Start by tracing the RID alias down the PCI topology as

@@ -123,56 +146,56 @@ static int __get_pci_rid(struct pci_dev *pdev, u16 alias, void *data)
 	 * bus into the system beyond, and which IOMMU it ends up at.
 	 */
 	iommu_spec.np = NULL;
-	if (of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
-			   "iommu-map-mask", &iommu_spec.np, iommu_spec.args))
-		return NULL;
+	err = of_pci_map_rid(bridge_np, iommu_spec.args[0], "iommu-map",
+			     "iommu-map-mask", &iommu_spec.np,
+			     iommu_spec.args);
+	if (err)
+		return err == -ENODEV ? NULL : ERR_PTR(err);
 
-	ops = iommu_ops_from_fwnode(&iommu_spec.np->fwnode);
-	if (!ops || !ops->of_xlate ||
-	    iommu_fwspec_init(&pdev->dev, &iommu_spec.np->fwnode, ops) ||
-	    ops->of_xlate(&pdev->dev, &iommu_spec))
-		ops = NULL;
+	ops = of_iommu_xlate(&pdev->dev, &iommu_spec);
 
 	of_node_put(iommu_spec.np);
 	return ops;
 }
 
-const struct iommu_ops *of_iommu_configure(struct device *dev,
-					   struct device_node *master_np)
+static const struct iommu_ops
+*of_platform_iommu_init(struct device *dev, struct device_node *np)
 {
 	struct of_phandle_args iommu_spec;
-	struct device_node *np;
 	const struct iommu_ops *ops = NULL;
 	int idx = 0;
 
-	if (dev_is_pci(dev))
-		return of_pci_iommu_configure(to_pci_dev(dev), master_np);
-
 	/*
 	 * We don't currently walk up the tree looking for a parent IOMMU.
 	 * See the `Notes:' section of
 	 * Documentation/devicetree/bindings/iommu/iommu.txt
 	 */
-	while (!of_parse_phandle_with_args(master_np, "iommus",
-					   "#iommu-cells", idx,
-					   &iommu_spec)) {
-		np = iommu_spec.np;
-		ops = iommu_ops_from_fwnode(&np->fwnode);
-
-		if (!ops || !ops->of_xlate ||
-		    iommu_fwspec_init(dev, &np->fwnode, ops) ||
-		    ops->of_xlate(dev, &iommu_spec))
-			goto err_put_node;
-
-		of_node_put(np);
+	while (!of_parse_phandle_with_args(np, "iommus", "#iommu-cells",
+					   idx, &iommu_spec)) {
+		ops = of_iommu_xlate(dev, &iommu_spec);
+		of_node_put(iommu_spec.np);
 		idx++;
+		if (IS_ERR_OR_NULL(ops))
+			break;
 	}
 
 	return ops;
+}
+
+const struct iommu_ops *of_iommu_configure(struct device *dev,
+					   struct device_node *master_np)
+{
+	const struct iommu_ops *ops;
+
+	if (!master_np)
+		return NULL;
+
+	if (dev_is_pci(dev))
+		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
+	else
+		ops = of_platform_iommu_init(dev, master_np);
 
-err_put_node:
-	of_node_put(np);
-	return NULL;
+	return IS_ERR(ops) ? NULL : ops;
 }
 
 static int __init of_iommu_init(void)

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

From: Robin Murphy <[email protected]>

IOMMU configuration represents unchanging properties of the hardware, and as such should only need happen once in a device's lifetime, but the necessary interaction with the IOMMU device and driver complicates exactly when that point should be.

Since the only reasonable tool available for handling the inter-device dependency is probe deferral, we need to prepare of_iommu_configure() to run later than it is currently called (i.e. at driver probe rather than device creation), to handle being retried, and to tell whether a not-yet present IOMMU should be waited for or skipped (by virtue of having declared a built-in driver or not).

---
 drivers/iommu/of_iommu.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)

index 8f4e599..c8be889 100644
@@ -96,6 +96,19 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 }
 EXPORT_SYMBOL_GPL(of_get_dma_window);
 
+static bool of_iommu_driver_present(struct device_node *np)
+{
+	/*
+	 * If the IOMMU still isn't ready by the time we reach init, assume
+	 * it never will be. We don't want to defer indefinitely, nor attempt
+	 * to dereference __iommu_of_table after it's been freed.
+	 */
+	if (system_state > SYSTEM_BOOTING)
+		return false;
+
+	return of_match_node(&__iommu_of_table, np);
+}
+
 static const struct iommu_ops
 *of_iommu_xlate(struct device *dev, struct of_phandle_args *iommu_spec)
 {

@@ -104,12 +117,20 @@ int of_get_dma_window(struct device_node *dn, const char *prefix, int index,
 	int err;
 
 	ops = iommu_ops_from_fwnode(fwnode);
-	if (!ops || !ops->of_xlate)
+	if ((ops && !ops->of_xlate) ||
+	    (!ops && !of_iommu_driver_present(iommu_spec->np)))
 		return NULL;
 
 	err = iommu_fwspec_init(dev, &iommu_spec->np->fwnode, ops);
 	if (err)
 		return ERR_PTR(err);
+	/*
+	 * The otherwise-empty fwspec handily serves to indicate the specific
+	 * IOMMU device we're waiting for, which will be useful if we ever get
+	 * a proper probe-ordering dependency mechanism in future.
+	 */
+	if (!ops)
+		return ERR_PTR(-EPROBE_DEFER);
 
 	err = ops->of_xlate(dev, iommu_spec);
 	if (err)

@@ -186,14 +207,34 @@ const struct iommu_ops *of_iommu_configure(struct device *dev,
 					   struct device_node *master_np)
 {
 	const struct iommu_ops *ops;
+	struct iommu_fwspec *fwspec = dev->iommu_fwspec;
 
 	if (!master_np)
 		return NULL;
 
+	if (fwspec) {
+		if (fwspec->ops)
+			return fwspec->ops;
+
+		/* In the deferred case, start again from scratch */
+		iommu_fwspec_free(dev);
+	}
+
 	if (dev_is_pci(dev))
 		ops = of_pci_iommu_init(to_pci_dev(dev), master_np);
 	else
 		ops = of_platform_iommu_init(dev, master_np);
+	/*
+	 * If we have reason to believe the IOMMU driver missed the initial
+	 * add_device callback for dev, replay it to get things in order.
+	 */
+	if (!IS_ERR_OR_NULL(ops) && ops->add_device &&
+	    dev->bus && !dev->iommu_group) {
+		int err = ops->add_device(dev);
+
+		if (err)
+			ops = ERR_PTR(err);
+	}
 
 	return IS_ERR(ops) ? NULL : ops;
 }

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/[email protected]>

Invalid dma-ranges values should be worked around when retrieving the DMA range in of_dma_get_range(), not by all callers of the function. This isn't much of a problem now that we have a single caller, but that situation will change when moving DMA configuration to device probe time.

---
 drivers/of/address.c | 20 ++++++++++++++++++--
 drivers/of/device.c  | 15 ---------------
 2 files changed, 18 insertions(+), 17 deletions(-)

index 02b2903..6aeb816 100644
@@ -819,8 +819,8 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  *	CPU addr (phys_addr_t)	: pna cells
  *	size			: nsize cells
  *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * Return 0 on success, -ENODEV if the "dma-ranges" property was not found for
+ * this device in DT, or -EINVAL if the CPU address or size is invalid.
  */
 int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
 {

@@ -880,6 +880,22 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
 	*dma_addr = dmaaddr;
 
 	*size = of_read_number(ranges + naddr + pna, nsize);
+	/*
+	 * DT nodes sometimes incorrectly set the size as a mask. Work around
+	 * those incorrect DT by computing the size as mask + 1.
+	 */
+	if (*size & 1) {
+		pr_warn("%s: size 0x%llx for dma-range in node(%s) set as mask\n",
+			__func__, *size, np->full_name);
+		*size = *size + 1;
+	}
+
+	if (!*size) {
+		pr_err("%s: invalid size zero for dma-range in node(%s)\n",
+		       __func__, np->full_name);
+		ret = -EINVAL;
+		goto out;
+	}
 
 	pr_debug("dma_addr(%llx) cpu_addr(%llx) size(%llx)\n",
 		 *dma_addr, *paddr, *size);

index b1e6beb..09dedd0 100644
@@ -110,21 +110,6 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 		size = dev->coherent_dma_mask + 1;
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
-
-		/*
-		 * Add a work around to treat the size as mask + 1 in case
-		 * it is defined in DT as a mask.
-		 */
-		if (size & 1) {
-			dev_warn(dev, "Invalid size 0x%llx for dma-range\n",
-				 size);
-			size = size + 1;
-		}
-
-		if (!size) {
-			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
-			return;
-		}
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Just for completeness:

Please drop this patch from the series. I just now sent Rob an alternative that reflects other changes that have occurred since this patch series was first created many, many moons ago
(https://lkml.org/lkml/2017/4/6/53).

Thanks,

Frank

Hi Frank,

ok, would drop this.

Regards,
Sricharan

"QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Hi Frank,

Fine by me too - if we can get rid of that dubious fixup altogether, all the better. Feel free to transfer my Reviewed-by to your patch if you like.

Thanks,
Robin.

From: Laurent Pinchart <laurent.pinchart+renesas-ryLnwIuWjnjg/[email protected]>

As part of moving DMA initializing to probe time the of_dma_deconfigure() function will need to be called from different source files. Make it public and move it to drivers/of/device.c where the of_dma_configure() function is.

---
 drivers/of/device.c       | 12 ++++++++++++
 drivers/of/platform.c     |  5 -----
 include/linux/of_device.h |  3 +++
 3 files changed, 15 insertions(+), 5 deletions(-)

index 09dedd0..c17c19d 100644
@@ -136,6 +136,18 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 }
 EXPORT_SYMBOL_GPL(of_dma_configure);
 
+/**
+ * of_dma_deconfigure - Clean up DMA configuration
+ * @dev:	Device for which to clean up DMA configuration
+ *
+ * Clean up all configuration performed by of_dma_configure_ops() and free all
+ * resources that have been allocated.
+ */
+void of_dma_deconfigure(struct device *dev)
+{
+	arch_teardown_dma_ops(dev);
+}
+
 int of_device_register(struct platform_device *pdev)
 {
 	device_initialize(&pdev->dev);

index 5dfcc96..5344db5 100644
@@ -158,11 +158,6 @@ struct platform_device *of_device_alloc(struct device_node *np,
 }
 EXPORT_SYMBOL(of_device_alloc);
 
-static void of_dma_deconfigure(struct device *dev)
-{
-	arch_teardown_dma_ops(dev);
-}
-
 /**
  * of_platform_device_create_pdata - Alloc, initialize and register an of_device
  * @np: pointer to node to create device for

index c12dace..af98455 100644
@@ -56,6 +56,7 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 
 void of_dma_configure(struct device *dev, struct device_node *np);
+void of_dma_deconfigure(struct device *dev);
 #else /* CONFIG_OF */
 
 static inline int of_driver_match_device(struct device *dev,

@@ -105,6 +106,8 @@ static inline struct device_node *of_cpu_device_node_get(int cpu)
 }
 static inline void of_dma_configure(struct device *dev, struct device_node *np)
 {}
+static inline void of_dma_deconfigure(struct device *dev)
+{}
 #endif /* CONFIG_OF */
 
 #endif /* _LINUX_OF_DEVICE_H */

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

From: Lorenzo Pieralisi <[email protected]>

The IOMMU probe deferral implementation requires a mechanism to detect if drivers for SMMU components are built-in in the kernel to detect whether IOMMU configuration for a given device should be deferred (ie SMMU drivers present but still not probed) or not (drivers not present).

Add a simple function to IORT to detect if SMMU drivers for SMMU components managed by IORT are built-in in the kernel.

---
 drivers/acpi/arm64/iort.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

index 4a5bb96..3dd9ec3 100644
@@ -523,6 +523,19 @@ static int arm_smmu_iort_xlate(struct device *dev, u32 streamid,
 	return ret;
 }
 
+static inline bool iort_iommu_driver_enabled(u8 type)
+{
+	switch (type) {
+	case ACPI_IORT_NODE_SMMU_V3:
+		return IS_BUILTIN(CONFIG_ARM_SMMU_V3);
+	case ACPI_IORT_NODE_SMMU:
+		return IS_BUILTIN(CONFIG_ARM_SMMU);
+	default:
+		pr_warn("IORT node type %u does not describe an SMMU\n", type);
+		return false;
+	}
+}
+
 static const struct iommu_ops *iort_iommu_xlate(struct device *dev,
 					struct acpi_iort_node *node,
 					u32 streamid)

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Ah, if only DT could be this neat and tidy :D

Size of the dma-range is calculated as coherent_dma_mask + 1 and passed to arch_setup_dma_ops further. It overflows when the coherent_dma_mask is set for full 64 bits 0xFFFFFFFFFFFFFFFF, resulting in size getting passed as 0 wrongly. Fix this by passsing in max(mask, mask + 1). Note that in this case when the mask is set to full 64bits, we will be passing the mask itself to arch_setup_dma_ops instead of the size. The real fix for this should be to make arch_setup_dma_ops receive the mask and handle it, to be done in the future.

---
 drivers/of/device.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index c17c19d..c2ae6bb 100644
@@ -107,7 +107,7 @@ void of_dma_configure(struct device *dev, struct device_node *np)
 	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
 	if (ret < 0) {
 		dma_addr = offset = 0;
-		size = dev->coherent_dma_mask + 1;
+		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 	} else {
 		offset = PFN_DOWN(paddr - dma_addr);
 		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);

QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

Agreed; in the meantime, this fix is neat and robust enough.

I'm not sure I agree there. Granted, there exist many more problematic aspects than are dealt with here (I've got more patches cooking to sort out some of the other issues we have with dma-ranges), but considering size specifically:

- It is not possible to explicitly specify a range with a size of 2^64 in DT. If someone does specify a size of 0, they've done a silly thing and should not be surprised that it ends badly.

- It *is* perfectly legitimate for bus code (or a previous device driver, once we start coming here at probe time) to have set a device's DMA mask to 0xffffffffffffffffULL. If this code then blindly overflows and infers an invalid size of 0 from that, breaking things in the process, that is this code's fault alone. It just so happens that nothing managed to trigger the latent problem until patch #7 here shakes up the callsites.

Yes, wacky impossible base + size combinations in DT were a theoretical problem before, and remain a theoretical problem, but also fall into the "how did you ever expect this to work?" category. There's certainly plenty more we can do to improve the DT parsing/validation, but that still doesn't apply to this path where the information is *not* coming from the DT at all.

Having started writing that patch too, I can tell you it's a big bugger touching multiple architectures and fixing up various drivers doing stupid things, hence why I'm happy with this point fix being the lesser of two evils in terms of not holding up this mostly-orthogonal series.

Robin.

And because of this, we allow ~0 (both 32 and 64 bit) in DT dma-ranges and fix these up as 2^32 and 2^64 sizes.

Rob

...which is what Frank's patch gets rid of.

Robin.

The existing code that uses size does not appear capable of dealing with the case of DMA mask of 0xffffffffffffffffULL since 2^64 does not fit into size.

The code affected by the DMA mask is not within my area of knowledge, so take the following with a grain of salt. If a DMA mask of 0xffffffffffffffffULL is provided, would the code still work without error (though with reduced capability) if the mask was changed to 0xefffffffffffffffULL? I would guess that the location to do so would be where dev->coherent_dma_mask is set, or some other location that is not of_dma_configure(). This would just be a temporary workaround.

Hi Frank,