Skip to contentSkip to conversation

Enable fw_devlink=on by default

PATCH In review Saravana Kannan 13 participants 89 messages Dec 18, 2020 – Feb 17, 2021 1 message not in the archive Page 1 of 2

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover Enable fw_devlink=on by default

As discussed in LPC 2020, cyclic dependencies in firmware that couldn't be broken using logic was one of the last remaining reasons fw_devlink=on couldn't be set by default.

This series changes fw_devlink so that when a cyclic dependency is found in firmware, the links between those devices fallback to permissive mode behavior. This way, the rest of the system still benefits from fw_devlink, but the ambiguous cases fallback to permissive mode.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

Greg/Rafael,

Can we get this pulled into 5.11-rc1 or -rc2 soon please? I expect to see some issues due to device drivers that aren't following best practices (they don't expose the device to driver core). Want to identify those early on and try to have them fixed before 5.11 release. See [1] for an example of such a case.

If we do end up have to revert anything, it'll just be Patch 5/5 (a one liner).

Marc,

You had hit issues with fw_devlink=on before on some of your systems. Want to give this a shot?

Jisheng,

Want to fix up one of those gpio drivers you were having problems with?

Thanks,
Saravana

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Cc Jisheng Zhang <[email protected]>
Cc Kevin Hilman <[email protected]>
Cc John Stultz <[email protected]>
Cc Nicolas Saenz Julienne <[email protected]>
Cc Marc Zyngier <[email protected]>

Saravana Kannan (5):
driver core: Add debug logs for device link related probe deferrals driver core: Add device link support for INFERRED flag driver core: Have fw_devlink use DL_FLAG_INFERRED
driver core: Handle cycles in device links created by fw_devlink driver core: Set fw_devlink=on by default

 drivers/base/core.c    | 101 +++++++++++++++++++++++++++++++++++------
 include/linux/device.h |   2 +
 2 files changed, 90 insertions(+), 13 deletions(-)
Signature · 1 line
2.29.2.684.gfbc64c5ab5-goog
#1 lore raw

PATCH 1/5 driver core: Add debug logs for device link related probe deferrals

There's insufficient logging when device links or fw_devlink (waiting to create device links) cause probe deferrals. This makes it hard to debug devices not getting probed. So, add debug logs to make it easy to debug.

---
 drivers/base/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/base/core.c +6 -0
index 25e08e5f40bd..fe8601197b84 100644
@@ -929,6 +929,10 @@ int device_links_check_suppliers(struct device *dev)
 	mutex_lock(&fwnode_link_lock);
 	if (dev->fwnode && !list_empty(&dev->fwnode->suppliers) &&
 	    !fw_devlink_is_permissive()) {
+		dev_dbg(dev, "probe deferral - wait for supplier %pfwP\n",
+			list_first_entry(&dev->fwnode->suppliers,
+			struct fwnode_link,
+			c_hook)->supplier);
 		mutex_unlock(&fwnode_link_lock);
 		return -EPROBE_DEFER;
 	}
@@ -943,6 +947,8 @@ int device_links_check_suppliers(struct device *dev)
 		if (link->status != DL_STATE_AVAILABLE &&
 		    !(link->flags & DL_FLAG_SYNC_STATE_ONLY)) {
 			device_links_missing_supplier(dev);
+			dev_dbg(dev, "probe deferral - supplier %s not ready\n",
+				dev_name(link->supplier));
 			ret = -EPROBE_DEFER;
 			break;
 		}
Signature · 1 line
2.29.2.684.gfbc64c5ab5-goog
#2 lore raw

PATCH 2/5 driver core: Add device link support for INFERRED flag

In reply to Saravana Kannan

This flag can never be added to a device link that already exists and doesn't have the flag set. It can only be added when a device link is created for the first time or it can be maintained if the device link already has the it set.

This flag will be used for marking device links created ONLY by inferring dependencies from data and NOT from explicit action by device drivers/frameworks. This will be useful in the future when we need to deal with cycles in dependencies inferred from firmware.

---
 drivers/base/core.c    | 15 +++++++++++----
 include/linux/device.h |  2 ++
 2 files changed, 13 insertions(+), 4 deletions(-)
drivers/base/core.c +11 -4
index fe8601197b84..5827dbff7f21 100644
@@ -229,7 +229,8 @@ int device_is_dependent(struct device *dev, void *target)
 		return ret;
 
 	list_for_each_entry(link, &dev->links.consumers, s_node) {
-		if (link->flags == (DL_FLAG_SYNC_STATE_ONLY | DL_FLAG_MANAGED))
+		if ((link->flags & ~DL_FLAG_INFERRED) ==
+		    (DL_FLAG_SYNC_STATE_ONLY | DL_FLAG_MANAGED))
 			continue;
 
 		if (link->consumer == target)
@@ -302,7 +303,8 @@ static int device_reorder_to_tail(struct device *dev, void *not_used)
 
 	device_for_each_child(dev, NULL, device_reorder_to_tail);
 	list_for_each_entry(link, &dev->links.consumers, s_node) {
-		if (link->flags == (DL_FLAG_SYNC_STATE_ONLY | DL_FLAG_MANAGED))
+		if ((link->flags & ~DL_FLAG_INFERRED) ==
+		    (DL_FLAG_SYNC_STATE_ONLY | DL_FLAG_MANAGED))
 			continue;
 		device_reorder_to_tail(link->consumer, NULL);
 	}
@@ -546,7 +548,8 @@ postcore_initcall(devlink_class_init);
 #define DL_MANAGED_LINK_FLAGS (DL_FLAG_AUTOREMOVE_CONSUMER | \
 			       DL_FLAG_AUTOREMOVE_SUPPLIER | \
 			       DL_FLAG_AUTOPROBE_CONSUMER  | \
-			       DL_FLAG_SYNC_STATE_ONLY)
+			       DL_FLAG_SYNC_STATE_ONLY | \
+			       DL_FLAG_INFERRED)
 
 #define DL_ADD_VALID_FLAGS (DL_MANAGED_LINK_FLAGS | DL_FLAG_STATELESS | \
 			    DL_FLAG_PM_RUNTIME | DL_FLAG_RPM_ACTIVE)
@@ -615,7 +618,7 @@ struct device_link *device_link_add(struct device *consumer,
 	if (!consumer || !supplier || flags & ~DL_ADD_VALID_FLAGS ||
 	    (flags & DL_FLAG_STATELESS && flags & DL_MANAGED_LINK_FLAGS) ||
 	    (flags & DL_FLAG_SYNC_STATE_ONLY &&
-	     flags != DL_FLAG_SYNC_STATE_ONLY) ||
+	     (flags & ~DL_FLAG_INFERRED) != DL_FLAG_SYNC_STATE_ONLY) ||
 	    (flags & DL_FLAG_AUTOPROBE_CONSUMER &&
 	     flags & (DL_FLAG_AUTOREMOVE_CONSUMER |
 		      DL_FLAG_AUTOREMOVE_SUPPLIER)))
@@ -671,6 +674,10 @@ struct device_link *device_link_add(struct device *consumer,
 		if (link->consumer != consumer)
 			continue;
 
+		if (link->flags & DL_FLAG_INFERRED &&
+		    !(flags & DL_FLAG_INFERRED))
+			link->flags &= ~DL_FLAG_INFERRED;
+
 		if (flags & DL_FLAG_PM_RUNTIME) {
 			if (!(link->flags & DL_FLAG_PM_RUNTIME)) {
 				pm_runtime_new_link(consumer);
include/linux/device.h +2 -0
index 89bb8b84173e..cb5eb2e58c25 100644
@@ -323,6 +323,7 @@ enum device_link_state {
  * AUTOPROBE_CONSUMER: Probe consumer driver automatically after supplier binds.
  * MANAGED: The core tracks presence of supplier/consumer drivers (internal).
  * SYNC_STATE_ONLY: Link only affects sync_state() behavior.
+ * INFERRED: Inferred from data (eg: firmware) and not from driver actions.
  */
 #define DL_FLAG_STATELESS		BIT(0)
 #define DL_FLAG_AUTOREMOVE_CONSUMER	BIT(1)
@@ -332,6 +333,7 @@ enum device_link_state {
 #define DL_FLAG_AUTOPROBE_CONSUMER	BIT(5)
 #define DL_FLAG_MANAGED			BIT(6)
 #define DL_FLAG_SYNC_STATE_ONLY		BIT(7)
+#define DL_FLAG_INFERRED		BIT(8)
 
 /**
  * enum dl_dev_state - Device driver presence tracking information.
Signature · 1 line
2.29.2.684.gfbc64c5ab5-goog
#3 lore raw

PATCH 3/5 driver core: Have fw_devlink use DL_FLAG_INFERRED

In reply to Saravana Kannan

This will be useful in identifying device links created only due to fw_devlink when we need to break cyclic dependencies due to fw_devlink.

---
 drivers/base/core.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)
drivers/base/core.c +14 -8
index 5827dbff7f21..1107d03aa6b3 100644
@@ -1450,7 +1450,14 @@ static void device_links_purge(struct device *dev)
 	device_links_write_unlock();
 }
 
-static u32 fw_devlink_flags = DL_FLAG_SYNC_STATE_ONLY;
+#define FW_DEVLINK_FLAGS_PERMISSIVE	(DL_FLAG_INFERRED | \
+					 DL_FLAG_SYNC_STATE_ONLY)
+#define FW_DEVLINK_FLAGS_ON		(DL_FLAG_INFERRED | \
+					 DL_FLAG_AUTOPROBE_CONSUMER)
+#define FW_DEVLINK_FLAGS_RPM		(FW_DEVLINK_FLAGS_ON | \
+					 DL_FLAG_PM_RUNTIME)
+
+static u32 fw_devlink_flags = FW_DEVLINK_FLAGS_PERMISSIVE;
 static int __init fw_devlink_setup(char *arg)
 {
 	if (!arg)
@@ -1459,12 +1466,11 @@ static int __init fw_devlink_setup(char *arg)
 	if (strcmp(arg, "off") == 0) {
 		fw_devlink_flags = 0;
 	} else if (strcmp(arg, "permissive") == 0) {
-		fw_devlink_flags = DL_FLAG_SYNC_STATE_ONLY;
+		fw_devlink_flags = FW_DEVLINK_FLAGS_PERMISSIVE;
 	} else if (strcmp(arg, "on") == 0) {
-		fw_devlink_flags = DL_FLAG_AUTOPROBE_CONSUMER;
+		fw_devlink_flags = FW_DEVLINK_FLAGS_ON;
 	} else if (strcmp(arg, "rpm") == 0) {
-		fw_devlink_flags = DL_FLAG_AUTOPROBE_CONSUMER |
-				   DL_FLAG_PM_RUNTIME;
+		fw_devlink_flags = FW_DEVLINK_FLAGS_RPM;
 	}
 	return 0;
 }
@@ -1477,7 +1483,7 @@ u32 fw_devlink_get_flags(void)
 
 static bool fw_devlink_is_permissive(void)
 {
-	return fw_devlink_flags == DL_FLAG_SYNC_STATE_ONLY;
+	return fw_devlink_flags == FW_DEVLINK_FLAGS_PERMISSIVE;
 }
 
 static void fw_devlink_parse_fwnode(struct fwnode_handle *fwnode)
@@ -1624,7 +1630,7 @@ static void __fw_devlink_link_to_consumers(struct device *dev)
 				con_dev = NULL;
 			} else {
 				own_link = false;
-				dl_flags = DL_FLAG_SYNC_STATE_ONLY;
+				dl_flags = FW_DEVLINK_FLAGS_PERMISSIVE;
 			}
 		}
 
@@ -1679,7 +1685,7 @@ static void __fw_devlink_link_to_suppliers(struct device *dev,
 	if (own_link)
 		dl_flags = fw_devlink_get_flags();
 	else
-		dl_flags = DL_FLAG_SYNC_STATE_ONLY;
+		dl_flags = FW_DEVLINK_FLAGS_PERMISSIVE;
 
 	list_for_each_entry_safe(link, tmp, &fwnode->suppliers, c_hook) {
 		int ret;
Signature · 1 line
2.29.2.684.gfbc64c5ab5-goog
#4 lore raw

PATCH 4/5 driver core: Handle cycles in device links created by fw_devlink

In reply to Saravana Kannan

Sometimes, firmware can have cyclic dependencies between devices. But one or more of those dependencies in the cycle are false dependencies that don't affect the probing of the device.

fw_devlink can detect some of these false dependencies using logic. But when it can't, we don't want to block probing of the devices in this cyclic dependency.

So, instead of using normal device links for the devices in this cycle, we need to switch to SYNC_STATE_ONLY device links between these devices. This is so that sync_state() callback correctness is still maintained while we allow these device to probe.

This is functionally similar to switching to fw_devlink=permissive just for the devices in the cycle.

---
 drivers/base/core.c | 58 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)
drivers/base/core.c +57 -1
index 1107d03aa6b3..4cc030361165 100644
@@ -1505,6 +1505,53 @@ static void fw_devlink_parse_fwtree(struct fwnode_handle *fwnode)
 		fw_devlink_parse_fwtree(child);
 }
 
+/**
+ * fw_devlink_relax_cycle - Convert cyclic links to SYNC_STATE_ONLY links
+ * @con: Device to check dependencies for.
+ * @sup: Device to check against.
+ *
+ * Check if @sup depends on @con or any device dependent on it (its child or
+ * its consumer etc).  When such a cyclic dependency is found, convert all
+ * device links created solely by fw_devlink into SYNC_STATE_ONLY device links.
+ * This is the equivalent of doing fw_devlink=permissive just between the
+ * devices in the cycle. We need to do this because, at this point, fw_devlink
+ * can't tell which of these dependencies is not a real dependency.
+ *
+ * Return 1 if a cycle is found. Otherwise, return 0.
+ */
+int fw_devlink_relax_cycle(struct device *con, void *sup)
+{
+	struct device_link *link;
+	int ret;
+
+	if (con == sup)
+		return 1;
+
+	ret = device_for_each_child(con, sup, fw_devlink_relax_cycle);
+	if (ret)
+		return ret;
+
+	list_for_each_entry(link, &con->links.consumers, s_node) {
+		if ((link->flags & ~DL_FLAG_INFERRED) ==
+		    (DL_FLAG_SYNC_STATE_ONLY | DL_FLAG_MANAGED))
+			continue;
+
+		if (!fw_devlink_relax_cycle(link->consumer, sup))
+			continue;
+
+		ret = 1;
+
+		if (!(link->flags & DL_FLAG_INFERRED))
+			continue;
+
+		pm_runtime_drop_link(link);
+		link->flags = DL_FLAG_MANAGED | FW_DEVLINK_FLAGS_PERMISSIVE;
+		dev_dbg(link->consumer, "Relaxing link with %s\n",
+			dev_name(link->supplier));
+	}
+	return ret;
+}
+
 /**
  * fw_devlink_create_devlink - Create a device link from a consumer to fwnode
  * @con - Consumer device for the device link
@@ -1536,8 +1583,17 @@ static int fw_devlink_create_devlink(struct device *con,
 		 * If this fails, it is due to cycles in device links.  Just
 		 * give up on this link and treat it as invalid.
 		 */
-		if (!device_link_add(con, sup_dev, flags))
+		if (!device_link_add(con, sup_dev, flags) &&
+		    !(flags & DL_FLAG_SYNC_STATE_ONLY)) {
+			dev_info(con, "Fixing up cyclic dependency with %s\n",
+				 dev_name(sup_dev));
+			device_links_write_lock();
+			fw_devlink_relax_cycle(con, sup_dev);
+			device_links_write_unlock();
+			device_link_add(con, sup_dev,
+					FW_DEVLINK_FLAGS_PERMISSIVE);
 			ret = -EINVAL;
+		}
 
 		goto out;
 	}
Signature · 1 line
2.29.2.684.gfbc64c5ab5-goog
#5 lore raw

Re: [PATCH v1 4/5] driver core: Handle cycles in device links created by fw_devlink

I love your patch! Perhaps something to improve:

Hi Saravana,

I love your patch! Perhaps something to improve:

[auto build test WARNING on driver-core/driver-core-testing] [also build test WARNING on linus/master next-20201217] [cannot apply to linux/master v5.10]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Saravana-Kannan/Enable-fw_devlink-on-by-default/20201218-112111
base:   https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git accefff5b547a9a1d959c7e76ad539bf2480e78b
config: i386-randconfig-s001-20201217 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
reproduce:
        # apt-get install sparse
        # sparse version: v0.6.3-184-g1b896707-dirty
        # https://github.com/0day-ci/linux/commit/7bdc87ea0400318d827410f454ec7e5fbaf470c3
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Saravana-Kannan/Enable-fw_devlink-on-by-default/20201218-112111
        git checkout 7bdc87ea0400318d827410f454ec7e5fbaf470c3
        # save the attached .config to linux build tree
        make W=1 C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=i386 

If you fix the issue, kindly add following tag as appropriate

"sparse warnings: (new ones prefixed by >>)"

>> drivers/base/core.c:1522:5: sparse: sparse: symbol 'fw_devlink_relax_cycle' was not declared. Should it be static?

Please review and possibly fold the followup patch.

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#6 lore raw

[RFC PATCH] driver core: fw_devlink_relax_cycle() can be static

In reply to Saravana Kannan
Reported-by: kernel test robot <[email protected]>
---
 core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/base/core.c +1 -1
index 4cc0303611650c..4e15193aafad6a 100644
@@ -1519,7 +1519,7 @@ static void fw_devlink_parse_fwtree(struct fwnode_handle *fwnode)
  *
  * Return 1 if a cycle is found. Otherwise, return 0.
  */
-int fw_devlink_relax_cycle(struct device *con, void *sup)
+static int fw_devlink_relax_cycle(struct device *con, void *sup)
 {
 	struct device_link *link;
 	int ret;
#7 lore raw

Re: [PATCH v1 4/5] driver core: Handle cycles in device links created by fw_devlink

In reply to Saravana Kannan
I love your patch! Perhaps something to improve:

Hi Saravana,

I love your patch! Perhaps something to improve:

[auto build test WARNING on driver-core/driver-core-testing] [also build test WARNING on linus/master next-20201217] [cannot apply to linux/master v5.10]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Saravana-Kannan/Enable-fw_devlink-on-by-default/20201218-112111
base:   https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git accefff5b547a9a1d959c7e76ad539bf2480e78b
config: mips-randconfig-r016-20201217 (attached as .config)
compiler: mips64-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/7bdc87ea0400318d827410f454ec7e5fbaf470c3
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Saravana-Kannan/Enable-fw_devlink-on-by-default/20201218-112111 git checkout 7bdc87ea0400318d827410f454ec7e5fbaf470c3
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=mips

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

>> drivers/base/core.c:1522:5: warning: no previous prototype for 'fw_devlink_relax_cycle' [-Wmissing-prototypes]

1522 | int fw_devlink_relax_cycle(struct device *con, void *sup)
| ^~~~~~~~~~~~~~~~~~~~~~

vim +/fw_devlink_relax_cycle +1522 drivers/base/core.c

1507
1508 /**
1509 * fw_devlink_relax_cycle - Convert cyclic links to SYNC_STATE_ONLY links
1510 * @con: Device to check dependencies for.
1511 * @sup: Device to check against.
1512 *
1513 * Check if @sup depends on @con or any device dependent on it (its child or
1514 * its consumer etc). When such a cyclic dependency is found, convert all
1515 * device links created solely by fw_devlink into SYNC_STATE_ONLY device links.
1516 * This is the equivalent of doing fw_devlink=permissive just between the
1517 * devices in the cycle. We need to do this because, at this point, fw_devlink
1518 * can't tell which of these dependencies is not a real dependency.
1519 *
1520 * Return 1 if a cycle is found. Otherwise, return 0.
1521 */

> 1522	int fw_devlink_relax_cycle(struct device *con, void *sup)

1523 {
1524 struct device_link *link;
1525 int ret;
1526
1527 if (con == sup)
1528 return 1;
1529
1530 ret = device_for_each_child(con, sup, fw_devlink_relax_cycle);
1531 if (ret)
1532 return ret;
1533
1534 list_for_each_entry(link, &con->links.consumers, s_node) {
1535 if ((link->flags & ~DL_FLAG_INFERRED) ==
1536 (DL_FLAG_SYNC_STATE_ONLY | DL_FLAG_MANAGED))
1537 continue;
1538
1539 if (!fw_devlink_relax_cycle(link->consumer, sup))
1540 continue;
1541
1542 ret = 1;
1543
1544 if (!(link->flags & DL_FLAG_INFERRED))
1545 continue;
1546
1547 pm_runtime_drop_link(link);
1548 link->flags = DL_FLAG_MANAGED | FW_DEVLINK_FLAGS_PERMISSIVE;
1549 dev_dbg(link->consumer, "Relaxing link with %s\n",
1550 dev_name(link->supplier));
1551 }
1552 return ret;
1553 }
1554

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#8 lore raw

Re: [PATCH v1 4/5] driver core: Handle cycles in device links created by fw_devlink

In reply to Saravana Kannan
I love your patch! Perhaps something to improve:

Hi Saravana,

I love your patch! Perhaps something to improve:

[auto build test WARNING on driver-core/driver-core-testing] [also build test WARNING on linus/master next-20201217] [cannot apply to linux/master v5.10]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Saravana-Kannan/Enable-fw_devlink-on-by-default/20201218-112111
base:   https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git accefff5b547a9a1d959c7e76ad539bf2480e78b
config: riscv-randconfig-r014-20201217 (attached as .config)
compiler: clang version 12.0.0 (https://github.com/llvm/llvm-project cee1e7d14f4628d6174b33640d502bff3b54ae45)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install riscv cross compiling tool for clang build
# apt-get install binutils-riscv64-linux-gnu
# https://github.com/0day-ci/linux/commit/7bdc87ea0400318d827410f454ec7e5fbaf470c3
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Saravana-Kannan/Enable-fw_devlink-on-by-default/20201218-112111 git checkout 7bdc87ea0400318d827410f454ec7e5fbaf470c3
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=riscv

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

                                                                           ~~~~~~~~~~ ^
   arch/riscv/include/asm/mmio.h:87:48: note: expanded from macro 'readb_cpu'
   #define readb_cpu(c)            ({ u8  __r = __raw_readb(c); __r; })
                                                            ^
   In file included from drivers/base/core.c:27:
   In file included from include/linux/netdevice.h:37:
   In file included from include/net/net_namespace.h:39:
   In file included from include/linux/skbuff.h:31:
   In file included from include/linux/dma-mapping.h:10:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/riscv/include/asm/io.h:149:
   include/asm-generic/io.h:564:9: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           return inw(addr);
                  ^~~~~~~~~
   arch/riscv/include/asm/io.h:56:76: note: expanded from macro 'inw'
   #define inw(c)          ({ u16 __v; __io_pbr(); __v = readw_cpu((void*)(PCI_IOBASE + (c))); __io_par(__v); __v; })
                                                                           ~~~~~~~~~~ ^
   arch/riscv/include/asm/mmio.h:88:76: note: expanded from macro 'readw_cpu'
   #define readw_cpu(c)            ({ u16 __r = le16_to_cpu((__force __le16)__raw_readw(c)); __r; })
                                                                                        ^
   include/uapi/linux/byteorder/little_endian.h:36:51: note: expanded from macro '__le16_to_cpu'
   #define __le16_to_cpu(x) ((__force __u16)(__le16)(x))
                                                     ^
   In file included from drivers/base/core.c:27:
   In file included from include/linux/netdevice.h:37:
   In file included from include/net/net_namespace.h:39:
   In file included from include/linux/skbuff.h:31:
   In file included from include/linux/dma-mapping.h:10:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/riscv/include/asm/io.h:149:
   include/asm-generic/io.h:572:9: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           return inl(addr);
                  ^~~~~~~~~
   arch/riscv/include/asm/io.h:57:76: note: expanded from macro 'inl'
   #define inl(c)          ({ u32 __v; __io_pbr(); __v = readl_cpu((void*)(PCI_IOBASE + (c))); __io_par(__v); __v; })
                                                                           ~~~~~~~~~~ ^
   arch/riscv/include/asm/mmio.h:89:76: note: expanded from macro 'readl_cpu'
   #define readl_cpu(c)            ({ u32 __r = le32_to_cpu((__force __le32)__raw_readl(c)); __r; })
                                                                                        ^
   include/uapi/linux/byteorder/little_endian.h:34:51: note: expanded from macro '__le32_to_cpu'
   #define __le32_to_cpu(x) ((__force __u32)(__le32)(x))
                                                     ^
   In file included from drivers/base/core.c:27:
   In file included from include/linux/netdevice.h:37:
   In file included from include/net/net_namespace.h:39:
   In file included from include/linux/skbuff.h:31:
   In file included from include/linux/dma-mapping.h:10:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/riscv/include/asm/io.h:149:
   include/asm-generic/io.h:580:2: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           outb(value, addr);
           ^~~~~~~~~~~~~~~~~
   arch/riscv/include/asm/io.h:59:68: note: expanded from macro 'outb'
   #define outb(v,c)       ({ __io_pbw(); writeb_cpu((v),(void*)(PCI_IOBASE + (c))); __io_paw(); })
                                                                 ~~~~~~~~~~ ^
   arch/riscv/include/asm/mmio.h:91:52: note: expanded from macro 'writeb_cpu'
   #define writeb_cpu(v, c)        ((void)__raw_writeb((v), (c)))
                                                             ^
   In file included from drivers/base/core.c:27:
   In file included from include/linux/netdevice.h:37:
   In file included from include/net/net_namespace.h:39:
   In file included from include/linux/skbuff.h:31:
   In file included from include/linux/dma-mapping.h:10:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/riscv/include/asm/io.h:149:
   include/asm-generic/io.h:588:2: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           outw(value, addr);
           ^~~~~~~~~~~~~~~~~
   arch/riscv/include/asm/io.h:60:68: note: expanded from macro 'outw'
   #define outw(v,c)       ({ __io_pbw(); writew_cpu((v),(void*)(PCI_IOBASE + (c))); __io_paw(); })
                                                                 ~~~~~~~~~~ ^
   arch/riscv/include/asm/mmio.h:92:76: note: expanded from macro 'writew_cpu'
   #define writew_cpu(v, c)        ((void)__raw_writew((__force u16)cpu_to_le16(v), (c)))
                                                                                     ^
   In file included from drivers/base/core.c:27:
   In file included from include/linux/netdevice.h:37:
   In file included from include/net/net_namespace.h:39:
   In file included from include/linux/skbuff.h:31:
   In file included from include/linux/dma-mapping.h:10:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/riscv/include/asm/io.h:149:
   include/asm-generic/io.h:596:2: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           outl(value, addr);
           ^~~~~~~~~~~~~~~~~
   arch/riscv/include/asm/io.h:61:68: note: expanded from macro 'outl'
   #define outl(v,c)       ({ __io_pbw(); writel_cpu((v),(void*)(PCI_IOBASE + (c))); __io_paw(); })
                                                                 ~~~~~~~~~~ ^
   arch/riscv/include/asm/mmio.h:93:76: note: expanded from macro 'writel_cpu'
   #define writel_cpu(v, c)        ((void)__raw_writel((__force u32)cpu_to_le32(v), (c)))
                                                                                     ^
   In file included from drivers/base/core.c:27:
   In file included from include/linux/netdevice.h:37:
   In file included from include/net/net_namespace.h:39:
   In file included from include/linux/skbuff.h:31:
   In file included from include/linux/dma-mapping.h:10:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/riscv/include/asm/io.h:149:
   include/asm-generic/io.h:1005:55: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           return (port > MMIO_UPPER_LIMIT) ? NULL : PCI_IOBASE + port;
                                                     ~~~~~~~~~~ ^
>> drivers/base/core.c:1522:5: warning: no previous prototype for function 'fw_devlink_relax_cycle' [-Wmissing-prototypes]
   int fw_devlink_relax_cycle(struct device *con, void *sup)
       ^

drivers/base/core.c:1522:1: note: declare 'static' if the function is not intended to be used outside of this translation unit int fw_devlink_relax_cycle(struct device *con, void *sup)
^
static
8 warnings generated.

vim +/fw_devlink_relax_cycle +1522 drivers/base/core.c

1507
1508 /**
1509 * fw_devlink_relax_cycle - Convert cyclic links to SYNC_STATE_ONLY links
1510 * @con: Device to check dependencies for.
1511 * @sup: Device to check against.
1512 *
1513 * Check if @sup depends on @con or any device dependent on it (its child or
1514 * its consumer etc). When such a cyclic dependency is found, convert all
1515 * device links created solely by fw_devlink into SYNC_STATE_ONLY device links.
1516 * This is the equivalent of doing fw_devlink=permissive just between the
1517 * devices in the cycle. We need to do this because, at this point, fw_devlink
1518 * can't tell which of these dependencies is not a real dependency.
1519 *
1520 * Return 1 if a cycle is found. Otherwise, return 0.
1521 */

> 1522	int fw_devlink_relax_cycle(struct device *con, void *sup)

1523 {
1524 struct device_link *link;
1525 int ret;
1526
1527 if (con == sup)
1528 return 1;
1529
1530 ret = device_for_each_child(con, sup, fw_devlink_relax_cycle);
1531 if (ret)
1532 return ret;
1533
1534 list_for_each_entry(link, &con->links.consumers, s_node) {
1535 if ((link->flags & ~DL_FLAG_INFERRED) ==
1536 (DL_FLAG_SYNC_STATE_ONLY | DL_FLAG_MANAGED))
1537 continue;
1538
1539 if (!fw_devlink_relax_cycle(link->consumer, sup))
1540 continue;
1541
1542 ret = 1;
1543
1544 if (!(link->flags & DL_FLAG_INFERRED))
1545 continue;
1546
1547 pm_runtime_drop_link(link);
1548 link->flags = DL_FLAG_MANAGED | FW_DEVLINK_FLAGS_PERMISSIVE;
1549 dev_dbg(link->consumer, "Relaxing link with %s\n",
1550 dev_name(link->supplier));
1551 }
1552 return ret;
1553 }
1554

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#9 lore raw

PATCH 5/5 driver core: Set fw_devlink=on by default

In reply to Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

---
 drivers/base/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/base/core.c +1 -1
index 4cc030361165..803bfa6eb823 100644
@@ -1457,7 +1457,7 @@ static void device_links_purge(struct device *dev)
 #define FW_DEVLINK_FLAGS_RPM		(FW_DEVLINK_FLAGS_ON | \
 					 DL_FLAG_PM_RUNTIME)
 
-static u32 fw_devlink_flags = FW_DEVLINK_FLAGS_PERMISSIVE;
+static u32 fw_devlink_flags = FW_DEVLINK_FLAGS_ON;
 static int __init fw_devlink_setup(char *arg)
 {
 	if (!arg)
Signature · 1 line
2.29.2.684.gfbc64c5ab5-goog
#10 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Hi Saravana,

Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

This patch landed recently in linux next-20210111 as commit e590474768f1 ("driver core: Set fw_devlink=on by default"). Sadly it breaks Exynos IOMMU operation, what causes lots of devices being deferred and not probed at all. I've briefly checked and noticed that exynos_sysmmu_probe() is never called after this patch. This is really strange for me, as the SYSMMU controllers on Exynos platform are regular platform devices registered by the OF code. The driver code is here:
drivers/iommu/exynos-iommu.c, example dts:
arch/arm/boot/dts/exynos3250.dtsi (compatible = "samsung,exynos-sysmmu").

Quoted 9 lines
---
  drivers/base/core.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/base/core.c +0 -0
index 4cc030361165..803bfa6eb823 100644

drivers/base/core.c static void device_links_purge(struct device *dev) · line 1463

Quoted 8 lines
drivers/base/core.c +1 -1
@@ -1457,7 +1457,7 @@ static void device_links_purge(struct device *dev)
  #define FW_DEVLINK_FLAGS_RPM		(FW_DEVLINK_FLAGS_ON | \
  					 DL_FLAG_PM_RUNTIME)
 
-static u32 fw_devlink_flags = FW_DEVLINK_FLAGS_PERMISSIVE;
+static u32 fw_devlink_flags = FW_DEVLINK_FLAGS_ON;
  static int __init fw_devlink_setup(char *arg)
  {
  	if (!arg)

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#11 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 25 lines · Marek Szyprowski
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it.  If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 9 lines

This patch landed recently in linux next-20210111 as commit e590474768f1 ("driver core: Set fw_devlink=on by default"). Sadly it breaks Exynos IOMMU operation, what causes lots of devices being deferred and not probed at all. I've briefly checked and noticed that exynos_sysmmu_probe() is never called after this patch. This is really strange for me, as the SYSMMU controllers on Exynos platform are regular platform devices registered by the OF code. The driver code is
here: drivers/iommu/exynos-iommu.c, example dts:
arch/arm/boot/dts/exynos3250.dtsi (compatible = "samsung,exynos-sysmmu").

Okay, I found the source of this problem. It is caused by Exynos power domain driver, which is not platform driver yet. I will post a patch, which converts it to the platform driver.

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#12 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 37 lines · Marek Szyprowski
Quoted 25 lines · Marek Szyprowski
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 9 lines

This patch landed recently in linux next-20210111 as commit e590474768f1 ("driver core: Set fw_devlink=on by default"). Sadly it breaks Exynos IOMMU operation, what causes lots of devices being deferred and not probed at all. I've briefly checked and noticed that exynos_sysmmu_probe() is never called after this patch. This is really strange for me, as the SYSMMU controllers on Exynos platform are regular platform devices registered by the OF code. The driver code is
here: drivers/iommu/exynos-iommu.c, example dts:
arch/arm/boot/dts/exynos3250.dtsi (compatible = "samsung,exynos-sysmmu").

Quoted 3 lines

Okay, I found the source of this problem. It is caused by Exynos power domain driver, which is not platform driver yet. I will post a patch, which converts it to the platform driver.

Thanks Marek! Hopefully the debug logs I added were sufficient to figure out the reason.

-Saravana

#13 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 36 lines · Saravana Kannan
Quoted 34 lines · Marek Szyprowski
Quoted 33 lines · Marek Szyprowski
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

This patch landed recently in linux next-20210111 as commit e590474768f1 ("driver core: Set fw_devlink=on by default"). Sadly it breaks Exynos IOMMU operation, what causes lots of devices being deferred and not probed at all. I've briefly checked and noticed that exynos_sysmmu_probe() is never called after this patch. This is really strange for me, as the SYSMMU controllers on Exynos platform are regular platform devices registered by the OF code. The driver code is
here: drivers/iommu/exynos-iommu.c, example dts:
arch/arm/boot/dts/exynos3250.dtsi (compatible = "samsung,exynos-sysmmu").

Quoted 5 lines
Quoted 3 lines

Okay, I found the source of this problem. It is caused by Exynos power domain driver, which is not platform driver yet. I will post a patch, which converts it to the platform driver.

Thanks Marek! Hopefully the debug logs I added were sufficient to figure out the reason.

Frankly, it took me a while to figure out that device core waits for the power domain devices. Maybe it would be possible to add some more debug messages or hints? Like the reason of the deferred probe in /sys/kernel/debug/devices_deferred ?

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#14 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 41 lines · Marek Szyprowski
Quoted 39 lines · Saravana Kannan
Quoted 37 lines · Marek Szyprowski
Quoted 24 lines · Marek Szyprowski
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 9 lines

This patch landed recently in linux next-20210111 as commit e590474768f1 ("driver core: Set fw_devlink=on by default"). Sadly it breaks Exynos IOMMU operation, what causes lots of devices being deferred and not probed at all. I've briefly checked and noticed that exynos_sysmmu_probe() is never called after this patch. This is really strange for me, as the SYSMMU controllers on Exynos platform are regular platform devices registered by the OF code. The driver code is
here: drivers/iommu/exynos-iommu.c, example dts:
arch/arm/boot/dts/exynos3250.dtsi (compatible = "samsung,exynos-sysmmu").

Okay, I found the source of this problem. It is caused by Exynos power domain driver, which is not platform driver yet. I will post a patch, which converts it to the platform driver.

Quoted 7 lines
Quoted 2 lines

Thanks Marek! Hopefully the debug logs I added were sufficient to figure out the reason.

Frankly, it took me a while to figure out that device core waits for the power domain devices. Maybe it would be possible to add some more debug messages or hints? Like the reason of the deferred probe in /sys/kernel/debug/devices_deferred ?

There's already a /sys/devices/.../<device>/waiting_for_supplier file that tells you if the device is waiting for a supplier device to be added. That file goes away once the device probes. If the file has 1, then it's waiting for the supplier device to be added (like your case). If it's 0, then the device is just waiting on one of the existing suppliers to probe. You can find the existing suppliers through /sys/devices/.../<device>/supplier:*/supplier. Also, flip these dev_dbg() to dev_info() if you need more details about deferred probing.

https://lore.kernel.org/lkml/[email protected]/

Hopefully this meets what you are looking for?

-Saravana

#15 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Hi Saravana,

Quoted 48 lines · Saravana Kannan
Quoted 40 lines · Marek Szyprowski
Quoted 39 lines · Saravana Kannan
Quoted 37 lines · Marek Szyprowski
Quoted 24 lines · Marek Szyprowski
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 9 lines

This patch landed recently in linux next-20210111 as commit e590474768f1 ("driver core: Set fw_devlink=on by default"). Sadly it breaks Exynos IOMMU operation, what causes lots of devices being deferred and not probed at all. I've briefly checked and noticed that exynos_sysmmu_probe() is never called after this patch. This is really strange for me, as the SYSMMU controllers on Exynos platform are regular platform devices registered by the OF code. The driver code is
here: drivers/iommu/exynos-iommu.c, example dts:
arch/arm/boot/dts/exynos3250.dtsi (compatible = "samsung,exynos-sysmmu").

Okay, I found the source of this problem. It is caused by Exynos power domain driver, which is not platform driver yet. I will post a patch, which converts it to the platform driver.

Quoted 6 lines
Quoted 2 lines

Thanks Marek! Hopefully the debug logs I added were sufficient to figure out the reason.

Frankly, it took me a while to figure out that device core waits for the power domain devices. Maybe it would be possible to add some more debug messages or hints? Like the reason of the deferred probe in /sys/kernel/debug/devices_deferred ?

Quoted 9 lines

There's already a /sys/devices/.../<device>/waiting_for_supplier file that tells you if the device is waiting for a supplier device to be added. That file goes away once the device probes. If the file has 1, then it's waiting for the supplier device to be added (like your case). If it's 0, then the device is just waiting on one of the existing suppliers to probe. You can find the existing suppliers through /sys/devices/.../<device>/supplier:*/supplier. Also, flip these dev_dbg() to dev_info() if you need more details about deferred probing.

Frankly speaking I doubt that anyone will find those. Even experienced developer might need some time to figure it out.

I expect that such information will be at least in the mentioned /sys/kernel/debug/devices_deferred file. We already have infrastructure for putting the deferred probe reason there, see dev_err_probe() function. Even such a simple change makes the debugging this issue much easier:

drivers/base/core.c +2 -1
index cd8e518fadd6..ceb5aed5a84c 100644
@@ -937,12 +937,13 @@ int device_links_check_suppliers(struct device *dev)
         mutex_lock(&fwnode_link_lock);
         if (dev->fwnode && !list_empty(&dev->fwnode->suppliers) &&
             !fw_devlink_is_permissive()) {
-               dev_dbg(dev, "probe deferral - wait for supplier %pfwP\n",
+               ret = dev_err_probe(dev, -EPROBE_DEFER,
+                       "probe deferral - wait for supplier %pfwP\n",

list_first_entry(&dev->fwnode->suppliers,
                        struct fwnode_link,
                        c_hook)->supplier);
                mutex_unlock(&fwnode_link_lock);
-               return -EPROBE_DEFER;
+               return ret;
        }
        mutex_unlock(&fwnode_link_lock);

@@ -955,9 +956,9 @@ int device_links_check_suppliers(struct device *dev)
                if (link->status != DL_STATE_AVAILABLE && !(link->flags & DL_FLAG_SYNC_STATE_ONLY)) { device_links_missing_supplier(dev);
-                       dev_dbg(dev, "probe deferral - supplier %s not ready\n",
+                       ret = dev_err_probe(dev, -EPROBE_DEFER, +                               "probe deferral - supplier %s not ready\n",
                                dev_name(link->supplier));
-                       ret = -EPROBE_DEFER;
                        break;
                }
                WRITE_ONCE(link->status, DL_STATE_CONSUMER_PROBE);

After such change:

# cat /sys/kernet/debug/devices_deferred
sound
13620000.sysmmu platform: probe deferral - supplier 
10023c40.power-domain not ready
13630000.sysmmu platform: probe deferral - supplier 
10023c40.power-domain not ready
12e20000.sysmmu platform: probe deferral - supplier 
10023c20.power-domain not ready
11a20000.sysmmu platform: probe deferral - supplier 
10023c00.power-domain not ready
11a30000.sysmmu platform: probe deferral - supplier 
10023c00.power-domain not ready
11a40000.sysmmu platform: probe deferral - supplier 
10023c00.power-domain not ready
11a50000.sysmmu platform: probe deferral - supplier 
10023c00.power-domain not ready
11a60000.sysmmu platform: probe deferral - supplier 
10023c00.power-domain not ready
11e20000.sysmmu platform: probe deferral - supplier 
10023c80.power-domain not ready
12d00000.hdmi   platform: probe deferral - supplier 
10023c20.power-domain not ready
10048000.clock-controller       platform: probe deferral - supplier 
10023ca0.power-domain not ready
12260000.sysmmu platform: probe deferral - supplier 
10048000.clock-controller not ready
12270000.sysmmu platform: probe deferral - supplier 
10048000.clock-controller not ready
122a0000.sysmmu platform: probe deferral - supplier 
10048000.clock-controller not ready
122b0000.sysmmu platform: probe deferral - supplier 
10048000.clock-controller not ready
123b0000.sysmmu platform: probe deferral - supplier 
10048000.clock-controller not ready
123c0000.sysmmu platform: probe deferral - supplier 
10048000.clock-controller not ready
12c10000.mixer  platform: probe deferral - supplier 
10023c20.power-domain not ready
13000000.gpu    platform: probe deferral - supplier 
10023c60.power-domain not ready

Probably the message can be adjusted a bit, this would significantly help me finding that is the source of the problem.

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#16 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 106 lines · Marek Szyprowski

Hi Saravana,

Quoted 48 lines · Saravana Kannan
Quoted 40 lines · Marek Szyprowski
Quoted 39 lines · Saravana Kannan
Quoted 37 lines · Marek Szyprowski
Quoted 24 lines · Marek Szyprowski
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 9 lines

This patch landed recently in linux next-20210111 as commit e590474768f1 ("driver core: Set fw_devlink=on by default"). Sadly it breaks Exynos IOMMU operation, what causes lots of devices being deferred and not probed at all. I've briefly checked and noticed that exynos_sysmmu_probe() is never called after this patch. This is really strange for me, as the SYSMMU controllers on Exynos platform are regular platform devices registered by the OF code. The driver code is
here: drivers/iommu/exynos-iommu.c, example dts:
arch/arm/boot/dts/exynos3250.dtsi (compatible = "samsung,exynos-sysmmu").

Okay, I found the source of this problem. It is caused by Exynos power domain driver, which is not platform driver yet. I will post a patch, which converts it to the platform driver.

Quoted 6 lines
Quoted 2 lines

Thanks Marek! Hopefully the debug logs I added were sufficient to figure out the reason.

Frankly, it took me a while to figure out that device core waits for the power domain devices. Maybe it would be possible to add some more debug messages or hints? Like the reason of the deferred probe in /sys/kernel/debug/devices_deferred ?

Quoted 9 lines

There's already a /sys/devices/.../<device>/waiting_for_supplier file that tells you if the device is waiting for a supplier device to be added. That file goes away once the device probes. If the file has 1, then it's waiting for the supplier device to be added (like your case). If it's 0, then the device is just waiting on one of the existing suppliers to probe. You can find the existing suppliers through /sys/devices/.../<device>/supplier:*/supplier. Also, flip these dev_dbg() to dev_info() if you need more details about deferred probing.

Frankly speaking I doubt that anyone will find those. Even experienced developer might need some time to figure it out.

I expect that such information will be at least in the mentioned /sys/kernel/debug/devices_deferred file. We already have infrastructure for putting the deferred probe reason there, see dev_err_probe() function. Even such a simple change makes the debugging this issue much easier:

drivers/base/core.c +5 -4
index cd8e518fadd6..ceb5aed5a84c 100644
@@ -937,12 +937,13 @@ int device_links_check_suppliers(struct device *dev)
         mutex_lock(&fwnode_link_lock);
         if (dev->fwnode && !list_empty(&dev->fwnode->suppliers) &&
             !fw_devlink_is_permissive()) {
-               dev_dbg(dev, "probe deferral - wait for supplier %pfwP\n",
+               ret = dev_err_probe(dev, -EPROBE_DEFER,
+                       "probe deferral - wait for supplier %pfwP\n",
 list_first_entry(&dev->fwnode->suppliers,
                         struct fwnode_link,
                         c_hook)->supplier);
                 mutex_unlock(&fwnode_link_lock);
-               return -EPROBE_DEFER;
+               return ret;
         }
         mutex_unlock(&fwnode_link_lock);
 
@@ -955,9 +956,9 @@ int device_links_check_suppliers(struct device *dev)
                 if (link->status != DL_STATE_AVAILABLE &&
                     !(link->flags & DL_FLAG_SYNC_STATE_ONLY)) {
                         device_links_missing_supplier(dev);
-                       dev_dbg(dev, "probe deferral - supplier %s not
 ready\n",
+                       ret = dev_err_probe(dev, -EPROBE_DEFER,
+                               "probe deferral - supplier %s not ready\n",
                                 dev_name(link->supplier));
-                       ret = -EPROBE_DEFER;
                         break;
                 }
                 WRITE_ONCE(link->status, DL_STATE_CONSUMER_PROBE);
Quoted 3 lines

After such change:

# cat /sys/kernet/debug/devices_deferred

Sweet! I wasn't aware of this file at all.

However, on a side note, one of my TODO items is to not add devices to the deferred probe list if they'll never probe yet (due to suppliers not having probed). On a board I tested on, it cut down really_probe() calls by 75%! So the probe attempt itself effectively happens in graph order (which I think is pretty cool). So that's going to conflict with this file. I'll have to see what to do about that.

Thanks for this pointer. Let me sit on this for 2 weeks and see how I can incorporate your suggestion while allowing for the above. And then I'll send out a patch. Does that work?

-Saravana

Quoted 49 lines

sound
13620000.sysmmu platform: probe deferral - supplier 10023c40.power-domain not ready 13630000.sysmmu platform: probe deferral - supplier 10023c40.power-domain not ready 12e20000.sysmmu platform: probe deferral - supplier 10023c20.power-domain not ready 11a20000.sysmmu platform: probe deferral - supplier 10023c00.power-domain not ready 11a30000.sysmmu platform: probe deferral - supplier 10023c00.power-domain not ready 11a40000.sysmmu platform: probe deferral - supplier 10023c00.power-domain not ready 11a50000.sysmmu platform: probe deferral - supplier 10023c00.power-domain not ready 11a60000.sysmmu platform: probe deferral - supplier 10023c00.power-domain not ready 11e20000.sysmmu platform: probe deferral - supplier 10023c80.power-domain not ready
12d00000.hdmi platform: probe deferral - supplier
10023c20.power-domain not ready
10048000.clock-controller platform: probe deferral - supplier
10023ca0.power-domain not ready 12260000.sysmmu platform: probe deferral - supplier 10048000.clock-controller not ready 12270000.sysmmu platform: probe deferral - supplier 10048000.clock-controller not ready 122a0000.sysmmu platform: probe deferral - supplier 10048000.clock-controller not ready 122b0000.sysmmu platform: probe deferral - supplier 10048000.clock-controller not ready 123b0000.sysmmu platform: probe deferral - supplier 10048000.clock-controller not ready 123c0000.sysmmu platform: probe deferral - supplier 10048000.clock-controller not ready 12c10000.mixer platform: probe deferral - supplier 10023c20.power-domain not ready
13000000.gpu platform: probe deferral - supplier
10023c60.power-domain not ready

Probably the message can be adjusted a bit, this would significantly help me finding that is the source of the problem.

Best regards

--
Marek Szyprowski, PhD
Samsung R&D Institute Poland

#17 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Hi Saravana,

Quoted 116 lines · Saravana Kannan
Quoted 102 lines · Marek Szyprowski
Quoted 48 lines · Saravana Kannan
Quoted 40 lines · Marek Szyprowski
Quoted 39 lines · Saravana Kannan
Quoted 37 lines · Marek Szyprowski
Quoted 24 lines · Marek Szyprowski
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://protect2.fireeye.com/v1/url?k=68f5d8ba-376ee1f5-68f453f5-0cc47a30d446-324e64700545ab93&q=1&e=fb455b9e-c8c7-40d0-8e3c-d9d3713d519b&u=https%3A%2F%2Flore.kernel.org%2Flkml%2FCAGETcx9PiX%3D%3DmLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw%40mail.gmail.com%2F

Quoted 9 lines

This patch landed recently in linux next-20210111 as commit e590474768f1 ("driver core: Set fw_devlink=on by default"). Sadly it breaks Exynos IOMMU operation, what causes lots of devices being deferred and not probed at all. I've briefly checked and noticed that exynos_sysmmu_probe() is never called after this patch. This is really strange for me, as the SYSMMU controllers on Exynos platform are regular platform devices registered by the OF code. The driver code is
here: drivers/iommu/exynos-iommu.c, example dts:
arch/arm/boot/dts/exynos3250.dtsi (compatible = "samsung,exynos-sysmmu").

Okay, I found the source of this problem. It is caused by Exynos power domain driver, which is not platform driver yet. I will post a patch, which converts it to the platform driver.

Quoted 6 lines
Quoted 2 lines

Thanks Marek! Hopefully the debug logs I added were sufficient to figure out the reason.

Frankly, it took me a while to figure out that device core waits for the power domain devices. Maybe it would be possible to add some more debug messages or hints? Like the reason of the deferred probe in /sys/kernel/debug/devices_deferred ?

Quoted 9 lines

There's already a /sys/devices/.../<device>/waiting_for_supplier file that tells you if the device is waiting for a supplier device to be added. That file goes away once the device probes. If the file has 1, then it's waiting for the supplier device to be added (like your case). If it's 0, then the device is just waiting on one of the existing suppliers to probe. You can find the existing suppliers through /sys/devices/.../<device>/supplier:*/supplier. Also, flip these dev_dbg() to dev_info() if you need more details about deferred probing.

Frankly speaking I doubt that anyone will find those. Even experienced developer might need some time to figure it out.

I expect that such information will be at least in the mentioned /sys/kernel/debug/devices_deferred file. We already have infrastructure for putting the deferred probe reason there, see dev_err_probe() function. Even such a simple change makes the debugging this issue much easier:

drivers/base/core.c +5 -4
index cd8e518fadd6..ceb5aed5a84c 100644
@@ -937,12 +937,13 @@ int device_links_check_suppliers(struct device *dev)
          mutex_lock(&fwnode_link_lock);
          if (dev->fwnode && !list_empty(&dev->fwnode->suppliers) &&
              !fw_devlink_is_permissive()) {
-               dev_dbg(dev, "probe deferral - wait for supplier %pfwP\n",
+               ret = dev_err_probe(dev, -EPROBE_DEFER,
+                       "probe deferral - wait for supplier %pfwP\n",
 list_first_entry(&dev->fwnode->suppliers,
                          struct fwnode_link,
                          c_hook)->supplier);
                  mutex_unlock(&fwnode_link_lock);
-               return -EPROBE_DEFER;
+               return ret;
          }
          mutex_unlock(&fwnode_link_lock);
 
@@ -955,9 +956,9 @@ int device_links_check_suppliers(struct device *dev)
                  if (link->status != DL_STATE_AVAILABLE &&
                      !(link->flags & DL_FLAG_SYNC_STATE_ONLY)) {
                          device_links_missing_supplier(dev);
-                       dev_dbg(dev, "probe deferral - supplier %s not
 ready\n",
+                       ret = dev_err_probe(dev, -EPROBE_DEFER,
+                               "probe deferral - supplier %s not ready\n",
                                  dev_name(link->supplier));
-                       ret = -EPROBE_DEFER;
                          break;
                  }
                  WRITE_ONCE(link->status, DL_STATE_CONSUMER_PROBE);
Quoted 3 lines

After such change:

# cat /sys/kernet/debug/devices_deferred

Sweet! I wasn't aware of this file at all.

However, on a side note, one of my TODO items is to not add devices to the deferred probe list if they'll never probe yet (due to suppliers not having probed). On a board I tested on, it cut down really_probe() calls by 75%! So the probe attempt itself effectively happens in graph order (which I think is pretty cool). So that's going to conflict with this file. I'll have to see what to do about that.

Quoted 3 lines

Thanks for this pointer. Let me sit on this for 2 weeks and see how I can incorporate your suggestion while allowing for the above. And then I'll send out a patch. Does that work?

Fine for me.

Even if you want to change the core not to probe devices that miss their suppliers (what's good imho), the 'devices_deferred' file might still contain all of them. For user it is just a list of devices that are not yet available in the system with the optional reasons for that.

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#18 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 124 lines · Marek Szyprowski

Hi Saravana,

Quoted 116 lines · Saravana Kannan
Quoted 102 lines · Marek Szyprowski
Quoted 48 lines · Saravana Kannan
Quoted 40 lines · Marek Szyprowski
Quoted 39 lines · Saravana Kannan
Quoted 37 lines · Marek Szyprowski
Quoted 24 lines · Marek Szyprowski
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://protect2.fireeye.com/v1/url?k=68f5d8ba-376ee1f5-68f453f5-0cc47a30d446-324e64700545ab93&q=1&e=fb455b9e-c8c7-40d0-8e3c-d9d3713d519b&u=https%3A%2F%2Flore.kernel.org%2Flkml%2FCAGETcx9PiX%3D%3DmLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw%40mail.gmail.com%2F

Quoted 9 lines

This patch landed recently in linux next-20210111 as commit e590474768f1 ("driver core: Set fw_devlink=on by default"). Sadly it breaks Exynos IOMMU operation, what causes lots of devices being deferred and not probed at all. I've briefly checked and noticed that exynos_sysmmu_probe() is never called after this patch. This is really strange for me, as the SYSMMU controllers on Exynos platform are regular platform devices registered by the OF code. The driver code is
here: drivers/iommu/exynos-iommu.c, example dts:
arch/arm/boot/dts/exynos3250.dtsi (compatible = "samsung,exynos-sysmmu").

Okay, I found the source of this problem. It is caused by Exynos power domain driver, which is not platform driver yet. I will post a patch, which converts it to the platform driver.

Quoted 6 lines
Quoted 2 lines

Thanks Marek! Hopefully the debug logs I added were sufficient to figure out the reason.

Frankly, it took me a while to figure out that device core waits for the power domain devices. Maybe it would be possible to add some more debug messages or hints? Like the reason of the deferred probe in /sys/kernel/debug/devices_deferred ?

Quoted 9 lines

There's already a /sys/devices/.../<device>/waiting_for_supplier file that tells you if the device is waiting for a supplier device to be added. That file goes away once the device probes. If the file has 1, then it's waiting for the supplier device to be added (like your case). If it's 0, then the device is just waiting on one of the existing suppliers to probe. You can find the existing suppliers through /sys/devices/.../<device>/supplier:*/supplier. Also, flip these dev_dbg() to dev_info() if you need more details about deferred probing.

Frankly speaking I doubt that anyone will find those. Even experienced developer might need some time to figure it out.

I expect that such information will be at least in the mentioned /sys/kernel/debug/devices_deferred file. We already have infrastructure for putting the deferred probe reason there, see dev_err_probe() function. Even such a simple change makes the debugging this issue much easier:

drivers/base/core.c +5 -4
index cd8e518fadd6..ceb5aed5a84c 100644
@@ -937,12 +937,13 @@ int device_links_check_suppliers(struct device *dev)
          mutex_lock(&fwnode_link_lock);
          if (dev->fwnode && !list_empty(&dev->fwnode->suppliers) &&
              !fw_devlink_is_permissive()) {
-               dev_dbg(dev, "probe deferral - wait for supplier %pfwP\n",
+               ret = dev_err_probe(dev, -EPROBE_DEFER,
+                       "probe deferral - wait for supplier %pfwP\n",
 list_first_entry(&dev->fwnode->suppliers,
                          struct fwnode_link,
                          c_hook)->supplier);
                  mutex_unlock(&fwnode_link_lock);
-               return -EPROBE_DEFER;
+               return ret;
          }
          mutex_unlock(&fwnode_link_lock);
 
@@ -955,9 +956,9 @@ int device_links_check_suppliers(struct device *dev)
                  if (link->status != DL_STATE_AVAILABLE &&
                      !(link->flags & DL_FLAG_SYNC_STATE_ONLY)) {
                          device_links_missing_supplier(dev);
-                       dev_dbg(dev, "probe deferral - supplier %s not
 ready\n",
+                       ret = dev_err_probe(dev, -EPROBE_DEFER,
+                               "probe deferral - supplier %s not ready\n",
                                  dev_name(link->supplier));
-                       ret = -EPROBE_DEFER;
                          break;
                  }
                  WRITE_ONCE(link->status, DL_STATE_CONSUMER_PROBE);
Quoted 3 lines

After such change:

# cat /sys/kernet/debug/devices_deferred

Sweet! I wasn't aware of this file at all.

However, on a side note, one of my TODO items is to not add devices to the deferred probe list if they'll never probe yet (due to suppliers not having probed). On a board I tested on, it cut down really_probe() calls by 75%! So the probe attempt itself effectively happens in graph order (which I think is pretty cool). So that's going to conflict with this file. I'll have to see what to do about that.

Quoted 3 lines

Thanks for this pointer. Let me sit on this for 2 weeks and see how I can incorporate your suggestion while allowing for the above. And then I'll send out a patch. Does that work?

Quoted 6 lines

Fine for me.

Even if you want to change the core not to probe devices that miss their suppliers (what's good imho), the 'devices_deferred' file might still contain all of them. For user it is just a list of devices that are not yet available in the system with the optional reasons for that.

Right, I understood that :) My point was that I'm assuming the debugfs file loops through the deferred devices list. But with my optimization, it won't find all the devices. So, we might need YET another list. :-(

-Saravana

#19 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

In reply to Saravana Kannan

Hi Saravana,

Quoted 50 lines · Saravana Kannan
Quoted 40 lines · Marek Szyprowski
Quoted 39 lines · Saravana Kannan
Quoted 37 lines · Marek Szyprowski
Quoted 24 lines · Marek Szyprowski
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 9 lines

This patch landed recently in linux next-20210111 as commit e590474768f1 ("driver core: Set fw_devlink=on by default"). Sadly it breaks Exynos IOMMU operation, what causes lots of devices being deferred and not probed at all. I've briefly checked and noticed that exynos_sysmmu_probe() is never called after this patch. This is really strange for me, as the SYSMMU controllers on Exynos platform are regular platform devices registered by the OF code. The driver code is
here: drivers/iommu/exynos-iommu.c, example dts:
arch/arm/boot/dts/exynos3250.dtsi (compatible = "samsung,exynos-sysmmu").

Okay, I found the source of this problem. It is caused by Exynos power domain driver, which is not platform driver yet. I will post a patch, which converts it to the platform driver.

Quoted 7 lines
Quoted 2 lines

Thanks Marek! Hopefully the debug logs I added were sufficient to figure out the reason.

Frankly, it took me a while to figure out that device core waits for the power domain devices. Maybe it would be possible to add some more debug messages or hints? Like the reason of the deferred probe in /sys/kernel/debug/devices_deferred ?

Quoted 9 lines

There's already a /sys/devices/.../<device>/waiting_for_supplier file that tells you if the device is waiting for a supplier device to be added. That file goes away once the device probes. If the file has 1, then it's waiting for the supplier device to be added (like your case). If it's 0, then the device is just waiting on one of the existing suppliers to probe. You can find the existing suppliers through /sys/devices/.../<device>/supplier:*/supplier. Also, flip these dev_dbg() to dev_info() if you need more details about deferred probing.

How are we supposed to check the contents of that file, if the system doesn't even boot into userspace with a ramdisk? All hardware drivers fail to probe. The only thing that works is "earlycon keep_bootcon", and kernel output just stops after a while.

Thanks for your suggestions!

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#20 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

In reply to Saravana Kannan

Hi Saravana, again ;)

Quoted 20 lines

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

This breaks (at least) probing of the PCIe controllers of my board. The driver in question is
drivers/pci/controller/dwc/pci-layerscape.c
I've also put the maintainers of this driver on CC. Looks like it uses a proper struct device. But it uses builtin_platform_driver_probe() and apparently it waits for the iommu which uses module_platform_driver(). Dunno if that will work together.

The board device tree can be found here:
arch/arm64/boot/dts/freescale/fsl-ls1028a-kontron-sl28-var3-ads2.dts

Attached is the log with enabled "probe deferral" messages enabled.

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.11.0-rc3-next-20210115-00013-g43ea1c90dcc8-dirty (mwalle@mwalle01) (aarch64-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0, GNU ld (GNU Binutils for Debian) 2.31.1) #357 SMP PREEMPT Sun Jan 17 23:46:11 CET 2021
[    0.000000] Machine model: Kontron SMARC-sAL28 (Single PHY) on SMARC Eval 2.0 carrier
[    0.000000] efi: UEFI not found.
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x00000020ffffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x20ff7d9200-0x20ff7dafff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000020ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000002080000000-0x00000020ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000020ffffffff]
[    0.000000] On node 0 totalpages: 1048576
[    0.000000]   DMA zone: 8192 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 524288 pages, LIFO batch:63
[    0.000000]   Normal zone: 8192 pages used for memmap
[    0.000000]   Normal zone: 524288 pages, LIFO batch:63
[    0.000000] cma: Reserved 32 MiB at 0x00000000fcc00000
[    0.000000] percpu: Embedded 31 pages/cpu s89752 r8192 d29032 u126976
[    0.000000] pcpu-alloc: s89752 r8192 d29032 u126976 alloc=31*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Spectre-v3a
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 1032192
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: debug root=/dev/mmcblk0p2 rootwait
[    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000f8c00000-0x00000000fcc00000] (64MB)
[    0.000000] Memory: 3987204K/4194304K available (14592K kernel code, 2024K rwdata, 5776K rodata, 4736K init, 848K bss, 174332K reserved, 32768K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] ftrace: allocating 51093 entries in 200 pages
[    0.000000] ftrace: allocated 200 pages with 3 groups
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] 	Rude variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 256 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006040000
[    0.000000] ITS [mem 0x06020000-0x0603ffff]
[    0.000000] ITS@0x0000000006020000: allocated 65536 Devices @2080180000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: using LPI property table @0x0000002080200000
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000002080210000
[    0.000000] random: get_random_bytes called from start_kernel+0x668/0x830 with crng_init=0
[    0.000000] arch_timer: cp15 timer(s) running at 25.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c40939b5, max_idle_ns: 440795202646 ns
[    0.000000] sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every 4398046511100ns
[    0.000120] Console: colour dummy device 80x25
[    0.000389] printk: console [tty0] enabled
[    0.000439] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000)
[    0.000454] pid_max: default: 32768 minimum: 301
[    0.000500] LSM: Security Framework initializing
[    0.000553] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.000587] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.001510] rcu: Hierarchical SRCU implementation.
[    0.001667] Platform MSI: gic-its@6020000 domain created
[    0.001745] PCI/MSI: /interrupt-controller@6000000/gic-its@6020000 domain created
[    0.001980] EFI services will not be available.
[    0.002073] smp: Bringing up secondary CPUs ...
[    0.002344] Detected PIPT I-cache on CPU1
[    0.002365] GICv3: CPU1: found redistributor 1 region 0:0x0000000006060000
[    0.002375] GICv3: CPU1: using allocated LPI pending table @0x0000002080220000
[    0.002405] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.002471] smp: Brought up 1 node, 2 CPUs
[    0.002499] SMP: Total of 2 processors activated.
[    0.002507] CPU features: detected: 32-bit EL0 Support
[    0.002515] CPU features: detected: CRC32 instructions
[    0.002524] CPU features: detected: 32-bit EL1 Support
[    0.011735] CPU: All CPU(s) started at EL2
[    0.011761] alternatives: patching kernel code
[    0.012508] devtmpfs: initialized
[    0.014832] KASLR disabled due to lack of seed
[    0.020761] DMA-API: preallocated 65536 debug entries
[    0.020786] DMA-API: debugging enabled by kernel config
[    0.020795] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.020815] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    0.021457] pinctrl core: initialized pinctrl subsystem
[    0.021715] *************************************************************
[    0.021722] **     NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE    **
[    0.021729] **                                                         **
[    0.021736] **  IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL  **
[    0.021742] **                                                         **
[    0.021749] ** This means that this kernel is built to expose internal **
[    0.021756] ** IOMMU data structures, which may compromise security on **
[    0.021762] ** your system.                                            **
[    0.021769] **                                                         **
[    0.021776] ** If you see this message and you are not debugging the   **
[    0.021782] ** kernel, report this immediately to your vendor!         **
[    0.021789] **                                                         **
[    0.021795] **     NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE    **
[    0.021802] *************************************************************
[    0.021884] DMI not present or invalid.
[    0.022167] NET: Registered protocol family 16
[    0.023001] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations
[    0.023114] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.023286] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.023317] audit: initializing netlink subsys (disabled)
[    0.023434] audit: type=2000 audit(0.020:1): state=initialized audit_enabled=0 res=1
[    0.023717] thermal_sys: Registered thermal governor 'step_wise'
[    0.023953] cpuidle: using governor menu
[    0.024076] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.024112] ASID allocator initialised with 65536 entries
[    0.024394] Serial: AMBA PL011 UART driver
[    0.026053] Machine: Kontron SMARC-sAL28 (Single PHY) on SMARC Eval 2.0 carrier
[    0.026065] SoC family: QorIQ LS1028A
[    0.026071] SoC ID: svr:0x870b0110, Revision: 1.0
[    0.038310] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.038332] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.038342] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.038351] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.039157] cryptd: max_cpu_qlen set to 1000
[    0.040473] ACPI: Interpreter disabled.
[    0.040536] platform 22c0000.dma-controller: probe deferral - supplier 5000000.iommu not ready
[    0.040768] iommu: Default domain type: Translated 
[    0.040845] vgaarb: loaded
[    0.041009] SCSI subsystem initialized
[    0.041097] libata version 3.00 loaded.
[    0.041222] usbcore: registered new interface driver usbfs
[    0.041248] usbcore: registered new interface driver hub
[    0.041273] usbcore: registered new device driver usb
[    0.041554] imx-i2c 2000000.i2c: can't get pinctrl, bus recovery not supported
[    0.041843] i2c i2c-0: IMX I2C adapter registered
[    0.042000] imx-i2c 2030000.i2c: can't get pinctrl, bus recovery not supported
[    0.042104] i2c i2c-1: IMX I2C adapter registered
[    0.042208] imx-i2c 2040000.i2c: can't get pinctrl, bus recovery not supported
[    0.042391] i2c i2c-2: IMX I2C adapter registered
[    0.042516] mc: Linux media interface: v0.10
[    0.042539] videodev: Linux video capture interface: v2.00
[    0.042566] pps_core: LinuxPPS API ver. 1 registered
[    0.042573] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.042588] PTP clock support registered
[    0.042664] EDAC MC: Ver: 3.0.0
[    0.043046] FPGA manager framework
[    0.043090] Advanced Linux Sound Architecture Driver Initialized.
[    0.043543] clocksource: Switched to clocksource arch_sys_counter
[    0.066221] VFS: Disk quotas dquot_6.6.0
[    0.066273] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.066402] pnp: PnP ACPI: disabled
[    0.074668] NET: Registered protocol family 2
[    0.074996] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[    0.075026] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.075122] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear)
[    0.075454] TCP: Hash tables configured (established 32768 bind 32768)
[    0.075571] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.075604] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.075708] NET: Registered protocol family 1
[    0.076019] RPC: Registered named UNIX socket transport module.
[    0.076031] RPC: Registered udp transport module.
[    0.076038] RPC: Registered tcp transport module.
[    0.076045] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.076055] PCI: CLS 0 bytes, default 64
[    0.076425] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    0.076625] kvm [1]: IPA Size Limit: 44 bits
[    0.077159] kvm [1]: GICv3: no GICV resource entry
[    0.077170] kvm [1]: disabling GICv2 emulation
[    0.077189] kvm [1]: GIC system register CPU interface enabled
[    0.077228] kvm [1]: vgic interrupt IRQ9
[    0.077288] kvm [1]: Hyp mode initialized successfully
[    0.088537] Initialise system trusted keyrings
[    0.088653] workingset: timestamp_bits=44 max_order=20 bucket_order=0
[    0.092096] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.092507] NFS: Registering the id_resolver key type
[    0.092532] Key type id_resolver registered
[    0.092540] Key type id_legacy registered
[    0.092590] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.092691] 9p: Installing v9fs 9p2000 file system support
[    0.123462] Key type asymmetric registered
[    0.123473] Asymmetric key parser 'x509' registered
[    0.123499] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    0.123510] io scheduler mq-deadline registered
[    0.123517] io scheduler kyber registered
[    0.124604] platform 1f0000000.pcie: probe deferral - supplier 5000000.iommu not ready
[    0.124746] platform 3400000.pcie: probe deferral - supplier 5000000.iommu not ready
[    0.124761] platform 3500000.pcie: probe deferral - supplier 5000000.iommu not ready
[    0.125145] EINJ: ACPI disabled.
[    0.128791] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.129769] 21c0500.serial: ttyS0 at MMIO 0x21c0500 (irq = 24, base_baud = 12500000) is a 16550A
[    1.276625] printk: console [ttyS0] enabled
[    1.281262] 21c0600.serial: ttyS1 at MMIO 0x21c0600 (irq = 24, base_baud = 12500000) is a 16550A
[    1.290396] platform 2270000.serial: probe deferral - supplier 22c0000.dma-controller not ready
[    1.299780] arm-smmu 5000000.iommu: probing hardware configuration...
[    1.306253] arm-smmu 5000000.iommu: SMMUv2 with:
[    1.310895] arm-smmu 5000000.iommu: 	stage 1 translation
[    1.316232] arm-smmu 5000000.iommu: 	stage 2 translation
[    1.321566] arm-smmu 5000000.iommu: 	nested translation
[    1.326815] arm-smmu 5000000.iommu: 	stream matching with 128 register groups
[    1.333984] arm-smmu 5000000.iommu: 	64 context banks (0 stage-2 only)
[    1.340542] arm-smmu 5000000.iommu: 	Supported page sizes: 0x61311000
[    1.347011] arm-smmu 5000000.iommu: 	Stage-1: 48-bit VA -> 48-bit IPA
[    1.353481] arm-smmu 5000000.iommu: 	Stage-2: 48-bit IPA -> 48-bit PA
[    1.364504] loop: module loaded
[    1.367804] at24 0-0050: supply vcc not found, using dummy regulator
[    1.375049] at24 0-0050: 4096 byte 24c32 EEPROM, writable, 32 bytes/write
[    1.381959] at24 1-0057: supply vcc not found, using dummy regulator
[    1.389165] at24 1-0057: 8192 byte 24c64 EEPROM, writable, 32 bytes/write
[    1.396063] at24 2-0050: supply vcc not found, using dummy regulator
[    1.403265] at24 2-0050: 4096 byte 24c32 EEPROM, writable, 32 bytes/write
[    1.424830] platform 2120000.spi: probe deferral - supplier 22c0000.dma-controller not ready
[    1.433801] spi-nor spi0.0: w25q32dw (4096 Kbytes)
[    1.438888] 8 fixed-partitions partitions found on MTD device 20c0000.spi
[    1.445729] Creating 8 MTD partitions on "20c0000.spi":
[    1.450984] 0x000000000000-0x000000010000 : "rcw"
[    1.459943] 0x000000010000-0x000000100000 : "failsafe bootloader"
[    1.467909] 0x000000100000-0x000000140000 : "failsafe DP firmware"
[    1.475904] 0x000000140000-0x0000001e0000 : "failsafe trusted firmware"
[    1.483901] 0x0000001e0000-0x000000200000 : "reserved"
[    1.491903] 0x000000200000-0x000000210000 : "configuration store"
[    1.499897] 0x000000210000-0x0000003e0000 : "bootloader"
[    1.507909] 0x0000003e0000-0x000000400000 : "bootloader environment"
[    1.516489] libphy: Fixed MDIO Bus: probed
[    1.521010] tun: Universal TUN/TAP device driver, 1.6
[    1.526180] CAN device driver interface
[    1.530778] thunder_xcv, ver 1.0
[    1.534051] thunder_bgx, ver 1.0
[    1.537311] nicpf, ver 1.0
[    1.540304] hclge is initializing
[    1.543646] hns3: Hisilicon Ethernet Network Driver for Hip08 Family - version
[    1.550901] hns3: Copyright (c) 2017 Huawei Corporation.
[    1.556270] igb: Intel(R) Gigabit Ethernet Network Driver
[    1.561694] igb: Copyright (c) 2007-2014 Intel Corporation.
[    1.567359] sky2: driver version 1.30
[    1.571307] VFIO - User Level meta-driver version: 0.3
[    1.576745] dwc3 3100000.usb: Adding to iommu group 0
[    1.582262] dwc3 3110000.usb: Adding to iommu group 1
[    1.588251] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.594825] ehci-pci: EHCI PCI platform driver
[    1.599307] ehci-platform: EHCI generic platform driver
[    1.604627] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.610844] ohci-pci: OHCI PCI platform driver
[    1.615324] ohci-platform: OHCI generic platform driver
[    1.620846] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[    1.626370] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
[    1.634228] xhci-hcd xhci-hcd.0.auto: hcc params 0x0220f66d hci version 0x100 quirks 0x0000000002010010
[    1.643698] xhci-hcd xhci-hcd.0.auto: irq 29, io mem 0x03100000
[    1.650061] hub 1-0:1.0: USB hub found
[    1.653850] hub 1-0:1.0: 1 port detected
[    1.657959] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[    1.663477] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
[    1.671174] xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0 SuperSpeed
[    1.677758] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    1.686134] hub 2-0:1.0: USB hub found
[    1.689917] hub 2-0:1.0: 1 port detected
[    1.694085] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
[    1.699607] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 3
[    1.707473] xhci-hcd xhci-hcd.1.auto: hcc params 0x0220f66d hci version 0x100 quirks 0x0000000002010010
[    1.716936] xhci-hcd xhci-hcd.1.auto: irq 30, io mem 0x03110000
[    1.723244] hub 3-0:1.0: USB hub found
[    1.727026] hub 3-0:1.0: 1 port detected
[    1.731114] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
[    1.736633] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 4
[    1.744329] xhci-hcd xhci-hcd.1.auto: Host supports USB 3.0 SuperSpeed
[    1.750919] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    1.759300] hub 4-0:1.0: USB hub found
[    1.763090] hub 4-0:1.0: 1 port detected
[    1.767271] usbcore: registered new interface driver usb-storage
[    1.774135] udc-core: couldn't find an available UDC - added [g_ether] to list of pending drivers
[    1.783054] udc-core: couldn't find an available UDC - added [g_mass_storage] to list of pending drivers
[    1.792577] udc-core: couldn't find an available UDC - added [g_serial] to list of pending drivers
[    1.801840] input: buttons1 as /devices/platform/buttons1/input/input0
[    1.809670] ftm-alarm 2800000.timer: registered as rtc1
[    1.815661] rtc-rv8803 0-0032: Voltage low, temperature compensation stopped.
[    1.822837] rtc-rv8803 0-0032: Voltage low, data loss detected.
[    1.829877] rtc-rv8803 0-0032: Voltage low, data is invalid.
[    1.835639] rtc-rv8803 0-0032: registered as rtc0
[    1.840966] rtc-rv8803 0-0032: Voltage low, data is invalid.
[    1.846654] rtc-rv8803 0-0032: hctosys: unable to read the hardware clock
[    1.853617] i2c /dev entries driver
[    1.868940] sp805-wdt c000000.watchdog: registration successful
[    1.875000] sp805-wdt c010000.watchdog: registration successful
[    1.882900] sl28cpld-wdt 2000000.i2c:sl28cpld@4a:watchdog@4: initial timeout 6 sec
[    1.890998] qoriq-cpufreq qoriq-cpufreq: Freescale QorIQ CPU frequency scaling driver
[    1.899245] sdhci: Secure Digital Host Controller Interface driver
[    1.905461] sdhci: Copyright(c) Pierre Ossman
[    1.909933] Synopsys Designware Multimedia Card Interface Driver
[    1.916222] sdhci-pltfm: SDHCI platform and OF driver helper
[    1.922189] sdhci-esdhc 2140000.mmc: Adding to iommu group 2
[    1.927922] sdhci-esdhc 2150000.mmc: Adding to iommu group 3
[    1.933859] ledtrig-cpu: registered to indicate activity on CPUs
[    1.940481] usbcore: registered new interface driver usbhid
[    1.946088] usbhid: USB HID core driver
[    1.951040] wm8904 2-001a: supply DCVDD not found, using dummy regulator
[    1.957849] wm8904 2-001a: supply DBVDD not found, using dummy regulator
[    1.959412] mmc0: SDHCI controller on 2150000.mmc [2150000.mmc] using ADMA
[    1.964608] wm8904 2-001a: supply AVDD not found, using dummy regulator
[    1.971498] mmc1: SDHCI controller on 2140000.mmc [2140000.mmc] using ADMA
[    1.978161] wm8904 2-001a: supply CPVDD not found, using dummy regulator
[    1.991795] wm8904 2-001a: supply MICVDD not found, using dummy regulator
[    2.000066] wm8904 2-001a: revision A
[    2.011006] platform f140000.audio-controller: probe deferral - supplier 22c0000.dma-controller not ready
[    2.020630] platform f150000.audio-controller: probe deferral - supplier 22c0000.dma-controller not ready
[    2.030314] drop_monitor: Initializing network drop monitor service
[    2.036934] NET: Registered protocol family 10
[    2.041875] Segment Routing with IPv6
[    2.045615] NET: Registered protocol family 17
[    2.050199] can: controller area network core
[    2.054614] NET: Registered protocol family 29
[    2.059081] can: raw protocol
[    2.062060] can: broadcast manager protocol
[    2.066276] can: netlink gateway - max_hops=1
[    2.070785] 9pnet: Installing 9P2000 support
[    2.071171] random: fast init done
[    2.075110] Key type dns_resolver registered
[    2.078504] usb 3-1: new high-speed USB device number 2 using xhci-hcd
[    2.082977] registered taskstats version 1
[    2.093467] Loading compiled-in X.509 certificates
[    2.096902] mmc0: new HS400 MMC card at address 0001
[    2.100484] fsl-edma 22c0000.dma-controller: Adding to iommu group 4
[    2.103805] mmcblk0: mmc0:0001 S0J58X 29.6 GiB 
[    2.110777] pci-host-generic 1f0000000.pcie: host bridge /soc/pcie@1f0000000 ranges:
[    2.114498] mmcblk0boot0: mmc0:0001 S0J58X partition 1 31.5 MiB
[    2.122014] pci-host-generic 1f0000000.pcie:      MEM 0x01f8000000..0x01f815ffff -> 0x0000000000
[    2.128314] mmcblk0boot1: mmc0:0001 S0J58X partition 2 31.5 MiB
[    2.136776] pci-host-generic 1f0000000.pcie:      MEM 0x01f8160000..0x01f81cffff -> 0x0000000000
[    2.147614] mmcblk0rpmb: mmc0:0001 S0J58X partition 3 4.00 MiB, chardev (241:0)
[    2.151551] pci-host-generic 1f0000000.pcie:      MEM 0x01f81d0000..0x01f81effff -> 0x0000000000
[    2.163146]  mmcblk0: p1 p2
[    2.167727] pci-host-generic 1f0000000.pcie:      MEM 0x01f81f0000..0x01f820ffff -> 0x0000000000
[    2.179375] pci-host-generic 1f0000000.pcie:      MEM 0x01f8210000..0x01f822ffff -> 0x0000000000
[    2.179619] mmc1: new ultra high speed SDR104 SDHC card at address 5048
[    2.188227] pci-host-generic 1f0000000.pcie:      MEM 0x01f8230000..0x01f824ffff -> 0x0000000000
[    2.203691] pci-host-generic 1f0000000.pcie:      MEM 0x01fc000000..0x01fc3fffff -> 0x0000000000
[    2.203959] mmcblk1: mmc1:5048 SD16G 14.4 GiB 
[    2.212582] pci-host-generic 1f0000000.pcie: ECAM at [mem 0x1f0000000-0x1f00fffff] for [bus 00]
[    2.225049] GPT:Primary header thinks Alt. header is not at the end of the disk.
[    2.225832] pci-host-generic 1f0000000.pcie: PCI host bridge to bus 0000:00
[    2.233193] GPT:266272 != 30253055
[    2.240157] pci_bus 0000:00: root bus resource [bus 00]
[    2.240164] pci_bus 0000:00: root bus resource [mem 0x1f8000000-0x1f815ffff] (bus address [0x00000000-0x0015ffff])
[    2.243589] GPT:Alternate GPT header not at the end of the disk.
[    2.248824] pci_bus 0000:00: root bus resource [mem 0x1f8160000-0x1f81cffff pref] (bus address [0x00000000-0x0006ffff])
[    2.259211] GPT:266272 != 30253055
[    2.265242] pci_bus 0000:00: root bus resource [mem 0x1f81d0000-0x1f81effff] (bus address [0x00000000-0x0001ffff])
[    2.276103] GPT: Use GNU Parted to correct GPT errors.
[    2.279483] pci_bus 0000:00: root bus resource [mem 0x1f81f0000-0x1f820ffff pref] (bus address [0x00000000-0x0001ffff])
[    2.279489] pci_bus 0000:00: root bus resource [mem 0x1f8210000-0x1f822ffff] (bus address [0x00000000-0x0001ffff])
[    2.279494] pci_bus 0000:00: root bus resource [mem 0x1f8230000-0x1f824ffff pref] (bus address [0x00000000-0x0001ffff])
[    2.279499] pci_bus 0000:00: root bus resource [mem 0x1fc000000-0x1fc3fffff] (bus address [0x00000000-0x003fffff])
[    2.279521] pci 0000:00:00.0: [1957:e100] type 00 class 0x020001
[    2.289971]  mmcblk1: p1
[    2.295081] pci 0000:00:00.0: BAR 0: [mem 0x1f8000000-0x1f803ffff 64bit] (from Enhanced Allocation, properties 0x0)
[    2.356551] pci 0000:00:00.0: BAR 2: [mem 0x1f8160000-0x1f816ffff 64bit pref] (from Enhanced Allocation, properties 0x1)
[    2.359017] hub 3-1:1.0: USB hub found
[    2.367482] pci 0000:00:00.0: VF BAR 0: [mem 0x1f81d0000-0x1f81dffff 64bit] (from Enhanced Allocation, properties 0x4)
[    2.371308] hub 3-1:1.0: 7 ports detected
[    2.381993] pci 0000:00:00.0: VF BAR 2: [mem 0x1f81f0000-0x1f81fffff 64bit pref] (from Enhanced Allocation, properties 0x3)
[    2.397218] pci 0000:00:00.0: PME# supported from D0 D3hot
[    2.402737] pci 0000:00:00.0: VF(n) BAR0 space: [mem 0x1f81d0000-0x1f81effff 64bit] (contains BAR0 for 2 VFs)
[    2.412699] pci 0000:00:00.0: VF(n) BAR2 space: [mem 0x1f81f0000-0x1f820ffff 64bit pref] (contains BAR2 for 2 VFs)
[    2.423260] pci 0000:00:00.1: [1957:e100] type 00 class 0x020001
[    2.429323] pci 0000:00:00.1: BAR 0: [mem 0x1f8040000-0x1f807ffff 64bit] (from Enhanced Allocation, properties 0x0)
[    2.439810] pci 0000:00:00.1: BAR 2: [mem 0x1f8170000-0x1f817ffff 64bit pref] (from Enhanced Allocation, properties 0x1)
[    2.450731] pci 0000:00:00.1: VF BAR 0: [mem 0x1f8210000-0x1f821ffff 64bit] (from Enhanced Allocation, properties 0x4)
[    2.461478] pci 0000:00:00.1: VF BAR 2: [mem 0x1f8230000-0x1f823ffff 64bit pref] (from Enhanced Allocation, properties 0x3)
[    2.472674] pci 0000:00:00.1: PME# supported from D0 D3hot
[    2.478193] pci 0000:00:00.1: VF(n) BAR0 space: [mem 0x1f8210000-0x1f822ffff 64bit] (contains BAR0 for 2 VFs)
[    2.488163] pci 0000:00:00.1: VF(n) BAR2 space: [mem 0x1f8230000-0x1f824ffff 64bit pref] (contains BAR2 for 2 VFs)
[    2.498689] pci 0000:00:00.2: [1957:e100] type 00 class 0x020001
[    2.504747] pci 0000:00:00.2: BAR 0: [mem 0x1f8080000-0x1f80bffff 64bit] (from Enhanced Allocation, properties 0x0)
[    2.515234] pci 0000:00:00.2: BAR 2: [mem 0x1f8180000-0x1f818ffff 64bit pref] (from Enhanced Allocation, properties 0x1)
[    2.526167] pci 0000:00:00.2: PME# supported from D0 D3hot
[    2.531802] pci 0000:00:00.3: [1957:ee01] type 00 class 0x088001
[    2.537868] pci 0000:00:00.3: BAR 0: [mem 0x1f8100000-0x1f811ffff 64bit] (from Enhanced Allocation, properties 0x0)
[    2.548357] pci 0000:00:00.3: BAR 2: [mem 0x1f8190000-0x1f819ffff 64bit pref] (from Enhanced Allocation, properties 0x1)
[    2.559290] pci 0000:00:00.3: PME# supported from D0 D3hot
[    2.564915] pci 0000:00:00.4: [1957:ee02] type 00 class 0x088001
[    2.570972] pci 0000:00:00.4: BAR 0: [mem 0x1f8120000-0x1f813ffff 64bit] (from Enhanced Allocation, properties 0x0)
[    2.581457] pci 0000:00:00.4: BAR 2: [mem 0x1f81a0000-0x1f81affff 64bit pref] (from Enhanced Allocation, properties 0x1)
[    2.592395] pci 0000:00:00.4: PME# supported from D0 D3hot
[    2.598030] pci 0000:00:00.5: [1957:eef0] type 00 class 0x020801
[    2.604092] pci 0000:00:00.5: BAR 0: [mem 0x1f8140000-0x1f815ffff 64bit] (from Enhanced Allocation, properties 0x0)
[    2.614583] pci 0000:00:00.5: BAR 2: [mem 0x1f81b0000-0x1f81bffff 64bit pref] (from Enhanced Allocation, properties 0x1)
[    2.625505] pci 0000:00:00.5: BAR 4: [mem 0x1fc000000-0x1fc3fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    2.636003] pci 0000:00:00.5: PME# supported from D0 D3hot
[    2.641641] pci 0000:00:00.6: [1957:e100] type 00 class 0x020001
[    2.647697] pci 0000:00:00.6: BAR 0: [mem 0x1f80c0000-0x1f80fffff 64bit] (from Enhanced Allocation, properties 0x0)
[    2.658184] pci 0000:00:00.6: BAR 2: [mem 0x1f81c0000-0x1f81cffff 64bit pref] (from Enhanced Allocation, properties 0x1)
[    2.669118] pci 0000:00:00.6: PME# supported from D0 D3hot
[    2.675519] pci 0000:00:1f.0: [1957:e001] type 00 class 0x080700
[    2.681596] OF: /soc/pcie@1f0000000: no msi-map translation for id 0xf8 on (null)
[    2.689371] fsl_enetc 0000:00:00.0: Adding to iommu group 5
[    2.707554] usb 3-1.6: new full-speed USB device number 3 using xhci-hcd
[    2.799556] fsl_enetc 0000:00:00.0: enabling device (0400 -> 0402)
[    2.805802] fsl_enetc 0000:00:00.0: no MAC address specified for SI1, using 82:f0:96:19:76:9c
[    2.814373] fsl_enetc 0000:00:00.0: no MAC address specified for SI2, using 5e:fb:ae:4d:83:1f
[    2.823365] libphy: Freescale ENETC MDIO Bus: probed
[    2.829583] libphy: Freescale ENETC internal MDIO Bus: probed
[    2.836011] fsl_enetc 0000:00:00.1: Adding to iommu group 6
[    2.841724] fsl_enetc 0000:00:00.1: device is disabled, skipping
[    2.847885] fsl_enetc 0000:00:00.2: Adding to iommu group 7
[    2.853568] fsl_enetc 0000:00:00.2: device is disabled, skipping
[    2.859709] fsl_enetc_mdio 0000:00:00.3: Adding to iommu group 8
[    2.872274] hid-generic 0003:064F:2AF9.0001: device has no listeners, quitting
[    2.971555] fsl_enetc_mdio 0000:00:00.3: enabling device (0400 -> 0402)
[    2.978403] libphy: FSL PCIe IE Central MDIO Bus: probed
[    2.983940] mscc_felix 0000:00:00.5: Adding to iommu group 9
[    2.989781] mscc_felix 0000:00:00.5: device is disabled, skipping
[    2.996017] fsl_enetc 0000:00:00.6: Adding to iommu group 10
[    3.001802] fsl_enetc 0000:00:00.6: device is disabled, skipping
[    3.007871] OF: /soc/pcie@1f0000000: no iommu-map translation for id 0xf8 on (null)
[    3.015686] pcieport 0000:00:1f.0: PME: Signaling with IRQ 123
[    3.021749] pcieport 0000:00:1f.0: AER: enabled with IRQ 123
[    3.028170] 2270000.serial: ttyLP2 at MMIO 0x2270000 (irq = 25, base_baud = 12500000) is a FSL_LPUART
[    3.038551] spi-nor spi1.0: at25sl321 (4096 Kbytes)
[    3.047761] asoc-simple-card sound: ASoC: no DMI vendor name!
[    3.056720] input: buttons0 as /devices/platform/buttons0/input/input1
[    3.063570] ALSA device list:
[    3.066547]   #0: f150000.audio-controller-wm8904-hifi
[    3.075682] EXT4-fs (mmcblk0p2): INFO: recovery required on readonly filesystem
[    3.083049] EXT4-fs (mmcblk0p2): write access will be enabled during recovery
[    3.095936] EXT4-fs (mmcblk0p2): recovery complete
[    3.102318] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[    3.103561] usb 3-1.3: new high-speed USB device number 4 using xhci-hcd
[    3.112153] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
[    3.125913] devtmpfs: mounted
[    3.133476] Freeing unused kernel memory: 4736K
[    3.147609] Missing param value! Expected 'debug=...value...'
[    3.153388] Missing param value! Expected 'rootwait=...value...'
[    3.159421] Run /sbin/init as init process
[    3.163540]   with arguments:
[    3.166513]     /sbin/init
[    3.169228]   with environment:
[    3.172378]     HOME=/
[    3.174741]     TERM=linux
[    3.205526] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null). Quota mode: none.
[    3.223479] usb-storage 3-1.3:1.0: USB Mass Storage device detected
[    3.230192] scsi host0: usb-storage 3-1.3:1.0
[    3.277936] udevd[139]: starting version 3.2.8
[    3.283246] random: udevd: uninitialized urandom read (16 bytes read)
[    3.290280] random: udevd: uninitialized urandom read (16 bytes read)
[    3.296805] random: udevd: uninitialized urandom read (16 bytes read)
[    3.304858] udevd[139]: specified group 'kvm' unknown
[    3.317735] udevd[140]: starting eudev-3.2.8
[    4.634039] scsi 0:0:0:0: Direct-Access     JetFlash Transcend 32GB   1100 PQ: 0 ANSI: 6
[    4.646337] sd 0:0:0:0: [sda] 61702144 512-byte logical blocks: (31.6 GB/29.4 GiB)
[    4.657126] sd 0:0:0:0: [sda] Write Protect is off
[    4.662029] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[    4.673461] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    4.710188]  sda: sda1
[    4.715321] sd 0:0:0:0: [sda] Attached SCSI removable disk
[    4.753103] fsl_enetc 0000:00:00.0 gbe0: renamed from eth0
[    6.006576] urandom_read: 3 callbacks suppressed
[    6.006587] random: dd: uninitialized urandom read (512 bytes read)
[    6.108830] fsl_enetc 0000:00:00.0 gbe0: PHY [0000:00:00.0:05] driver [Qualcomm Atheros AR8031/AR8033] (irq=POLL)
[    6.127032] fsl_enetc 0000:00:00.0 gbe0: configuring for inband/sgmii link mode
[    6.153943] random: dropbear: uninitialized urandom read (32 bytes read)
[   10.212329] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow control rx/tx
[   10.220233] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes ready
[  164.963579] random: crng init done

HTH,
-michael

#21 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 2 lines · Michael Walle

Hi Saravana, again ;)

Hi again! :)

Quoted 24 lines
Quoted 21 lines

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 7 lines

This breaks (at least) probing of the PCIe controllers of my board. The driver in question is
drivers/pci/controller/dwc/pci-layerscape.c
I've also put the maintainers of this driver on CC. Looks like it uses a proper struct device. But it uses builtin_platform_driver_probe() and apparently it waits for the iommu which uses module_platform_driver(). Dunno if that will work together.

Yeah, the builtin vs module doesn't matter. I've had fw_devlink work multiple times with the consumer driver being built in and the supplier actually loaded as a module. Making that work is one of the goals of fw_devlink.

Quoted 4 lines

The board device tree can be found here:
arch/arm64/boot/dts/freescale/fsl-ls1028a-kontron-sl28-var3-ads2.dts

Attached is the log with enabled "probe deferral" messages enabled.

I took a look at the logs. As you said, pci seems to be waiting on iommu, but it's not clear why the iommu didn't probe by then. Can you add initcall_debug=1 and enable the logs in device_link_add()? Btw, I realize one compromise on the logs is to send them as an attachment instead of inline. That way, it's still archived in the list, but I don't have to deal with log lines getting wrapped, etc.

Thanks for reporting the issues. Also, could you try picking up all of these changes and giving it a shot. It's unlikely to help, but I want to rule out issues related to fixes in progress.

https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

Thanks,
Saravana

#22 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Am 2021-01-18 22:01, schrieb Saravana Kannan:

Quoted 32 lines · Michael Walle
Quoted 23 lines
Quoted 20 lines

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 7 lines

This breaks (at least) probing of the PCIe controllers of my board. The driver in question is
drivers/pci/controller/dwc/pci-layerscape.c
I've also put the maintainers of this driver on CC. Looks like it uses a proper struct device. But it uses builtin_platform_driver_probe() and apparently it waits for the iommu which uses module_platform_driver(). Dunno if that will work together.

Quoted 4 lines

Yeah, the builtin vs module doesn't matter. I've had fw_devlink work multiple times with the consumer driver being built in and the supplier actually loaded as a module. Making that work is one of the goals of fw_devlink.

Ok.

Quoted 12 lines
Quoted 4 lines

The board device tree can be found here:
arch/arm64/boot/dts/freescale/fsl-ls1028a-kontron-sl28-var3-ads2.dts

Attached is the log with enabled "probe deferral" messages enabled.

I took a look at the logs. As you said, pci seems to be waiting on iommu, but it's not clear why the iommu didn't probe by then. Can you add initcall_debug=1 and enable the logs in device_link_add()? Btw, I realize one compromise on the logs is to send them as an attachment instead of inline. That way, it's still archived in the list, but I don't have to deal with log lines getting wrapped, etc.

Quoted 7 lines

Thanks for reporting the issues. Also, could you try picking up all of these changes and giving it a shot. It's unlikely to help, but I want to rule out issues related to fixes in progress.

https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

Did pick them up, the last one had a conflict due some superfluous lines. Maybe they got reordered in that arrray.

Issue still persist. I've enabled the debug in device_link_add(), in device_links_check_suppliers() and booted with initcall_debug. Please see attached log. Lets see how that goes ;)

[    0.132687] calling  ls_pcie_driver_init+0x0/0x38 @ 1
[    0.132762] platform 3400000.pcie: probe deferral - supplier 5000000.iommu not ready
[    0.132777] platform 3500000.pcie: probe deferral - supplier 5000000.iommu not ready
[    0.132818] initcall ls_pcie_driver_init+0x0/0x38 returned -19 after 119 usecs

After that, ls_pcie_driver_init() is never called again.

-michael

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.11.0-rc4-next-20210119-00011-g04c7947f0b7e-dirty (mwalle@mwalle01) (aarch64-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0, GNU ld (GNU Binutils for Debian) 2.31.1) #359 SMP PREEMPT Tue Jan 19 09:22:23 CET 2021
[    0.000000] Machine model: Kontron KBox A-230-LS
[    0.000000] efi: UEFI not found.
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x00000020ffffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x20ff7d9200-0x20ff7dafff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000020ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000002080000000-0x00000020ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000020ffffffff]
[    0.000000] On node 0 totalpages: 1048576
[    0.000000]   DMA zone: 8192 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 524288 pages, LIFO batch:63
[    0.000000]   Normal zone: 8192 pages used for memmap
[    0.000000]   Normal zone: 524288 pages, LIFO batch:63
[    0.000000] cma: Reserved 32 MiB at 0x00000000fcc00000
[    0.000000] percpu: Embedded 31 pages/cpu s89752 r8192 d29032 u126976
[    0.000000] pcpu-alloc: s89752 r8192 d29032 u126976 alloc=31*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Spectre-v3a
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 1032192
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: debug root=/dev/mmcblk0p2 rootwait initcall_debug
[    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000f8c00000-0x00000000fcc00000] (64MB)
[    0.000000] Memory: 3987140K/4194304K available (14592K kernel code, 2024K rwdata, 5780K rodata, 4800K init, 848K bss, 174396K reserved, 32768K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] ftrace: allocating 51181 entries in 200 pages
[    0.000000] ftrace: allocated 200 pages with 3 groups
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] 	Rude variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 256 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006040000
[    0.000000] ITS [mem 0x06020000-0x0603ffff]
[    0.000000] ITS@0x0000000006020000: allocated 65536 Devices @2080180000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: using LPI property table @0x0000002080200000
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000002080210000
[    0.000000] random: get_random_bytes called from start_kernel+0x668/0x830 with crng_init=0
[    0.000000] arch_timer: cp15 timer(s) running at 25.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x5c40939b5, max_idle_ns: 440795202646 ns
[    0.000000] sched_clock: 56 bits at 25MHz, resolution 40ns, wraps every 4398046511100ns
[    0.000093] calling  con_init+0x0/0x220 @ 0
[    0.000130] Console: colour dummy device 80x25
[    0.000406] printk: console [tty0] enabled
[    0.000413] initcall con_init+0x0/0x220 returned 0 after 0 usecs
[    0.000424] calling  hvc_console_init+0x0/0x34 @ 0
[    0.000435] initcall hvc_console_init+0x0/0x34 returned 0 after 0 usecs
[    0.000445] calling  xen_cons_init+0x0/0x88 @ 0
[    0.000459] initcall xen_cons_init+0x0/0x88 returned 0 after 0 usecs
[    0.000470] calling  univ8250_console_init+0x0/0x50 @ 0
[    0.000482] initcall univ8250_console_init+0x0/0x50 returned 0 after 0 usecs
[    0.000527] Calibrating delay loop (skipped), value calculated using timer frequency.. 50.00 BogoMIPS (lpj=100000)
[    0.000543] pid_max: default: 32768 minimum: 301
[    0.000590] LSM: Security Framework initializing
[    0.000642] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.000677] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.001546] calling  trace_init_flags_sys_enter+0x0/0x2c @ 1
[    0.001570] initcall trace_init_flags_sys_enter+0x0/0x2c returned 0 after 0 usecs
[    0.001584] calling  trace_init_flags_sys_exit+0x0/0x2c @ 1
[    0.001595] initcall trace_init_flags_sys_exit+0x0/0x2c returned 0 after 0 usecs
[    0.001607] calling  cpu_suspend_init+0x0/0x58 @ 1
[    0.001619] initcall cpu_suspend_init+0x0/0x58 returned 0 after 0 usecs
[    0.001630] calling  asids_init+0x0/0xc0 @ 1
[    0.001650] initcall asids_init+0x0/0xc0 returned 0 after 0 usecs
[    0.001660] calling  xen_guest_init+0x0/0x348 @ 1
[    0.001671] initcall xen_guest_init+0x0/0x348 returned 0 after 0 usecs
[    0.001683] calling  spawn_ksoftirqd+0x0/0x58 @ 1
[    0.001742] initcall spawn_ksoftirqd+0x0/0x58 returned 0 after 0 usecs
[    0.001757] calling  migration_init+0x0/0x64 @ 1
[    0.001766] initcall migration_init+0x0/0x64 returned 0 after 0 usecs
[    0.001776] calling  srcu_bootup_announce+0x0/0x50 @ 1
[    0.001786] rcu: Hierarchical SRCU implementation.
[    0.001793] initcall srcu_bootup_announce+0x0/0x50 returned 0 after 0 usecs
[    0.001804] calling  rcu_spawn_core_kthreads+0x0/0xcc @ 1
[    0.001814] initcall rcu_spawn_core_kthreads+0x0/0xcc returned 0 after 0 usecs
[    0.001826] calling  rcu_spawn_gp_kthread+0x0/0x16c @ 1
[    0.001879] initcall rcu_spawn_gp_kthread+0x0/0x16c returned 0 after 0 usecs
[    0.001893] calling  check_cpu_stall_init+0x0/0x3c @ 1
[    0.001904] initcall check_cpu_stall_init+0x0/0x3c returned 0 after 0 usecs
[    0.001915] calling  rcu_sysrq_init+0x0/0x48 @ 1
[    0.001925] initcall rcu_sysrq_init+0x0/0x48 returned 0 after 0 usecs
[    0.001935] calling  cpu_stop_init+0x0/0xd8 @ 1
[    0.001998] initcall cpu_stop_init+0x0/0xd8 returned 0 after 0 usecs
[    0.002012] calling  init_events+0x0/0x68 @ 1
[    0.002039] initcall init_events+0x0/0x68 returned 0 after 0 usecs
[    0.002051] calling  init_trace_printk+0x0/0x28 @ 1
[    0.002063] initcall init_trace_printk+0x0/0x28 returned 0 after 0 usecs
[    0.002075] calling  event_trace_enable_again+0x0/0x38 @ 1
[    0.002087] initcall event_trace_enable_again+0x0/0x38 returned 0 after 0 usecs
[    0.002100] calling  jump_label_init_module+0x0/0x2c @ 1
[    0.002109] initcall jump_label_init_module+0x0/0x2c returned 0 after 0 usecs
[    0.002119] calling  initialize_ptr_random+0x0/0x70 @ 1
[    0.002131] initcall initialize_ptr_random+0x0/0x70 returned 0 after 0 usecs
[    0.002143] calling  its_pmsi_init+0x0/0xb8 @ 1
[    0.002183] Platform MSI: gic-its@6020000 domain created
[    0.002247] initcall its_pmsi_init+0x0/0xb8 returned 0 after 0 usecs
[    0.002259] calling  its_pci_msi_init+0x0/0xd4 @ 1
[    0.002285] PCI/MSI: /interrupt-controller@6000000/gic-its@6020000 domain created
[    0.002326] initcall its_pci_msi_init+0x0/0xd4 returned 0 after 0 usecs
[    0.002338] calling  brcmstb_soc_device_early_init+0x0/0xb0 @ 1
[    0.002515] initcall brcmstb_soc_device_early_init+0x0/0xb0 returned 0 after 0 usecs
[    0.002529] calling  brcmstb_biuctrl_init+0x0/0x468 @ 1
[    0.002570] initcall brcmstb_biuctrl_init+0x0/0x468 returned 0 after 0 usecs
[    0.002582] calling  efi_memreserve_root_init+0x0/0x48 @ 1
[    0.002594] initcall efi_memreserve_root_init+0x0/0x48 returned 0 after 0 usecs
[    0.002606] calling  arm_enable_runtime_services+0x0/0x1e4 @ 1
[    0.002617] EFI services will not be available.
[    0.002624] initcall arm_enable_runtime_services+0x0/0x1e4 returned 0 after 0 usecs
[    0.002636] calling  efi_earlycon_remap_fb+0x0/0x84 @ 1
[    0.002647] initcall efi_earlycon_remap_fb+0x0/0x84 returned 0 after 0 usecs
[    0.002659] calling  dummy_timer_register+0x0/0x44 @ 1
[    0.002672] initcall dummy_timer_register+0x0/0x44 returned 0 after 0 usecs
[    0.002771] smp: Bringing up secondary CPUs ...
[    0.003048] Detected PIPT I-cache on CPU1
[    0.003069] GICv3: CPU1: found redistributor 1 region 0:0x0000000006060000
[    0.003079] GICv3: CPU1: using allocated LPI pending table @0x0000002080220000
[    0.003107] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.003176] smp: Brought up 1 node, 2 CPUs
[    0.003204] SMP: Total of 2 processors activated.
[    0.003212] CPU features: detected: 32-bit EL0 Support
[    0.003221] CPU features: detected: CRC32 instructions
[    0.003230] CPU features: detected: 32-bit EL1 Support
[    0.012372] CPU: All CPU(s) started at EL2
[    0.012398] alternatives: patching kernel code
[    0.013127] devtmpfs: initialized
[    0.015219] calling  bpf_jit_charge_init+0x0/0x4c @ 1
[    0.015251] initcall bpf_jit_charge_init+0x0/0x4c returned 0 after 0 usecs
[    0.015265] calling  ipc_ns_init+0x0/0x4c @ 1
[    0.015280] initcall ipc_ns_init+0x0/0x4c returned 0 after 0 usecs
[    0.015291] calling  init_mmap_min_addr+0x0/0x28 @ 1
[    0.015301] initcall init_mmap_min_addr+0x0/0x28 returned 0 after 0 usecs
[    0.015311] calling  pci_realloc_setup_params+0x0/0x54 @ 1
[    0.015324] initcall pci_realloc_setup_params+0x0/0x54 returned 0 after 0 usecs
[    0.015337] calling  net_ns_init+0x0/0x164 @ 1
[    0.015501] initcall net_ns_init+0x0/0x164 returned 0 after 0 usecs
[    0.015517] calling  inet_frag_wq_init+0x0/0x5c @ 1
[    0.015587] initcall inet_frag_wq_init+0x0/0x5c returned 0 after 0 usecs
[    0.015639] calling  fpsimd_init+0x0/0xd4 @ 1
[    0.015652] initcall fpsimd_init+0x0/0xd4 returned 0 after 0 usecs
[    0.015663] calling  tagged_addr_init+0x0/0x40 @ 1
[    0.015679] initcall tagged_addr_init+0x0/0x40 returned 0 after 0 usecs
[    0.015690] calling  enable_mrs_emulation+0x0/0x34 @ 1
[    0.015701] initcall enable_mrs_emulation+0x0/0x34 returned 0 after 0 usecs
[    0.015712] calling  kaslr_init+0x0/0x84 @ 1
[    0.015723] KASLR disabled due to lack of seed
[    0.015729] initcall kaslr_init+0x0/0x84 returned 0 after 0 usecs
[    0.015740] calling  map_entry_trampoline+0x0/0x118 @ 1
[    0.015764] initcall map_entry_trampoline+0x0/0x118 returned 0 after 0 usecs
[    0.015776] calling  alloc_frozen_cpus+0x0/0x18 @ 1
[    0.015788] initcall alloc_frozen_cpus+0x0/0x18 returned 0 after 0 usecs
[    0.015799] calling  cpu_hotplug_pm_sync_init+0x0/0x34 @ 1
[    0.015811] initcall cpu_hotplug_pm_sync_init+0x0/0x34 returned 0 after 0 usecs
[    0.015824] calling  wq_sysfs_init+0x0/0x50 @ 1
[    0.015868] initcall wq_sysfs_init+0x0/0x50 returned 0 after 0 usecs
[    0.015880] calling  ksysfs_init+0x0/0xc8 @ 1
[    0.015901] initcall ksysfs_init+0x0/0xc8 returned 0 after 0 usecs
[    0.015911] calling  schedutil_gov_init+0x0/0x2c @ 1
[    0.015921] initcall schedutil_gov_init+0x0/0x2c returned 0 after 0 usecs
[    0.015931] calling  pm_init+0x0/0x88 @ 1
[    0.015968] initcall pm_init+0x0/0x88 returned 0 after 0 usecs
[    0.015979] calling  pm_disk_init+0x0/0x34 @ 1
[    0.015993] initcall pm_disk_init+0x0/0x34 returned 0 after 0 usecs
[    0.016004] calling  swsusp_header_init+0x0/0x48 @ 1
[    0.016014] initcall swsusp_header_init+0x0/0x48 returned 0 after 0 usecs
[    0.016024] calling  rcu_set_runtime_mode+0x0/0x34 @ 1
[    0.016035] initcall rcu_set_runtime_mode+0x0/0x34 returned 0 after 0 usecs
[    0.016045] calling  dma_init_reserved_memory+0x0/0x74 @ 1
[    0.016056] initcall dma_init_reserved_memory+0x0/0x74 returned -12 after 0 usecs
[    0.016068] calling  dma_debug_init+0x0/0x220 @ 1
[    0.021866] DMA-API: preallocated 65536 debug entries
[    0.021890] DMA-API: debugging enabled by kernel config
[    0.021898] initcall dma_debug_init+0x0/0x220 returned 0 after 3906 usecs
[    0.021920] calling  init_jiffies_clocksource+0x0/0x34 @ 1
[    0.021934] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.021948] initcall init_jiffies_clocksource+0x0/0x34 returned 0 after 0 usecs
[    0.021960] calling  futex_init+0x0/0x114 @ 1
[    0.021975] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    0.022009] initcall futex_init+0x0/0x114 returned 0 after 0 usecs
[    0.022020] calling  cgroup_wq_init+0x0/0x48 @ 1
[    0.022044] initcall cgroup_wq_init+0x0/0x48 returned 0 after 0 usecs
[    0.022055] calling  cgroup1_wq_init+0x0/0x48 @ 1
[    0.022069] initcall cgroup1_wq_init+0x0/0x48 returned 0 after 0 usecs
[    0.022081] calling  ftrace_mod_cmd_init+0x0/0x28 @ 1
[    0.022094] initcall ftrace_mod_cmd_init+0x0/0x28 returned 0 after 0 usecs
[    0.022105] calling  init_graph_trace+0x0/0x88 @ 1
[    0.022124] initcall init_graph_trace+0x0/0x88 returned 0 after 0 usecs
[    0.022136] calling  cpu_pm_init+0x0/0x30 @ 1
[    0.022150] initcall cpu_pm_init+0x0/0x30 returned 0 after 0 usecs
[    0.022161] calling  init_zero_pfn+0x0/0x38 @ 1
[    0.022170] initcall init_zero_pfn+0x0/0x38 returned 0 after 0 usecs
[    0.022180] calling  mem_cgroup_swap_init+0x0/0x80 @ 1
[    0.022201] initcall mem_cgroup_swap_init+0x0/0x80 returned 0 after 0 usecs
[    0.022212] calling  memory_failure_init+0x0/0xb0 @ 1
[    0.022222] initcall memory_failure_init+0x0/0xb0 returned 0 after 0 usecs
[    0.022233] calling  cma_init_reserved_areas+0x0/0x194 @ 1
[    0.022705] initcall cma_init_reserved_areas+0x0/0x194 returned 0 after 0 usecs
[    0.022719] calling  fsnotify_init+0x0/0x64 @ 1
[    0.022758] initcall fsnotify_init+0x0/0x64 returned 0 after 0 usecs
[    0.022770] calling  filelock_init+0x0/0xdc @ 1
[    0.022792] initcall filelock_init+0x0/0xdc returned 0 after 0 usecs
[    0.022804] calling  init_script_binfmt+0x0/0x34 @ 1
[    0.022815] initcall init_script_binfmt+0x0/0x34 returned 0 after 0 usecs
[    0.022827] calling  init_elf_binfmt+0x0/0x34 @ 1
[    0.022838] initcall init_elf_binfmt+0x0/0x34 returned 0 after 0 usecs
[    0.022849] calling  init_compat_elf_binfmt+0x0/0x34 @ 1
[    0.022860] initcall init_compat_elf_binfmt+0x0/0x34 returned 0 after 0 usecs
[    0.022872] calling  configfs_init+0x0/0xc4 @ 1
[    0.022894] initcall configfs_init+0x0/0xc4 returned 0 after 0 usecs
[    0.022906] calling  debugfs_init+0x0/0x98 @ 1
[    0.022917] initcall debugfs_init+0x0/0x98 returned 0 after 0 usecs
[    0.022927] calling  tracefs_init+0x0/0x5c @ 1
[    0.022938] initcall tracefs_init+0x0/0x5c returned 0 after 0 usecs
[    0.022948] calling  securityfs_init+0x0/0xa0 @ 1
[    0.023017] initcall securityfs_init+0x0/0xa0 returned 0 after 0 usecs
[    0.023032] calling  prandom_init_early+0x0/0x12c @ 1
[    0.023043] initcall prandom_init_early+0x0/0x12c returned 0 after 0 usecs
[    0.023053] calling  pinctrl_init+0x0/0xd0 @ 1
[    0.023065] pinctrl core: initialized pinctrl subsystem
[    0.023096] initcall pinctrl_init+0x0/0xd0 returned 0 after 0 usecs
[    0.023109] calling  gpiolib_dev_init+0x0/0x14c @ 1
[    0.023180] initcall gpiolib_dev_init+0x0/0x14c returned 0 after 0 usecs
[    0.023194] calling  fsl_guts_init+0x0/0x30 @ 1
[    0.023220] initcall fsl_guts_init+0x0/0x30 returned 0 after 0 usecs
[    0.023232] calling  virtio_init+0x0/0x40 @ 1
[    0.023254] initcall virtio_init+0x0/0x40 returned 0 after 0 usecs
[    0.023265] calling  regulator_init+0x0/0xc4 @ 1
[    0.023424] initcall regulator_init+0x0/0xc4 returned 0 after 0 usecs
[    0.023437] calling  iommu_init+0x0/0x50 @ 1
[    0.023453] 
[    0.023459] *************************************************************
[    0.023466] **     NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE    **
[    0.023473] **                                                         **
[    0.023480] **  IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL  **
[    0.023487] **                                                         **
[    0.023494] ** This means that this kernel is built to expose internal **
[    0.023500] ** IOMMU data structures, which may compromise security on **
[    0.023507] ** your system.                                            **
[    0.023514] **                                                         **
[    0.023520] ** If you see this message and you are not debugging the   **
[    0.023527] ** kernel, report this immediately to your vendor!         **
[    0.023534] **                                                         **
[    0.023540] **     NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE    **
[    0.023547] *************************************************************
[    0.023554] initcall iommu_init+0x0/0x50 returned 0 after 0 usecs
[    0.023564] calling  component_debug_init+0x0/0x3c @ 1
[    0.023578] initcall component_debug_init+0x0/0x3c returned 0 after 0 usecs
[    0.023590] calling  genpd_bus_init+0x0/0x30 @ 1
[    0.023611] initcall genpd_bus_init+0x0/0x30 returned 0 after 0 usecs
[    0.023622] calling  soc_bus_register+0x0/0x64 @ 1
[    0.023642] initcall soc_bus_register+0x0/0x64 returned 0 after 0 usecs
[    0.023653] calling  register_cpufreq_notifier+0x0/0x70 @ 1
[    0.023664] initcall register_cpufreq_notifier+0x0/0x70 returned -22 after 0 usecs
[    0.023676] calling  opp_debug_init+0x0/0x3c @ 1
[    0.023697] initcall opp_debug_init+0x0/0x3c returned 0 after 0 usecs
[    0.023709] calling  cpufreq_core_init+0x0/0x88 @ 1
[    0.023723] initcall cpufreq_core_init+0x0/0x88 returned 0 after 0 usecs
[    0.023733] calling  cpufreq_gov_performance_init+0x0/0x2c @ 1
[    0.023745] initcall cpufreq_gov_performance_init+0x0/0x2c returned 0 after 0 usecs
[    0.023756] calling  cpufreq_gov_userspace_init+0x0/0x2c @ 1
[    0.023767] initcall cpufreq_gov_userspace_init+0x0/0x2c returned 0 after 0 usecs
[    0.023778] calling  CPU_FREQ_GOV_ONDEMAND_init+0x0/0x30 @ 1
[    0.023789] initcall CPU_FREQ_GOV_ONDEMAND_init+0x0/0x30 returned 0 after 0 usecs
[    0.023800] calling  cpufreq_dt_platdev_init+0x0/0x150 @ 1
[    0.023837] initcall cpufreq_dt_platdev_init+0x0/0x150 returned -19 after 0 usecs
[    0.023849] calling  cpuidle_init+0x0/0x40 @ 1
[    0.023867] initcall cpuidle_init+0x0/0x40 returned 0 after 0 usecs
[    0.023878] calling  capsule_reboot_register+0x0/0x30 @ 1
[    0.023890] initcall capsule_reboot_register+0x0/0x30 returned 0 after 0 usecs
[    0.023902] calling  arm_dmi_init+0x0/0x40 @ 1
[    0.023914] DMI not present or invalid.
[    0.023920] initcall arm_dmi_init+0x0/0x40 returned 0 after 0 usecs
[    0.023931] calling  sock_init+0x0/0xc4 @ 1
[    0.024163] initcall sock_init+0x0/0xc4 returned 0 after 3906 usecs
[    0.024177] calling  net_inuse_init+0x0/0x40 @ 1
[    0.024191] initcall net_inuse_init+0x0/0x40 returned 0 after 0 usecs
[    0.024201] calling  net_defaults_init+0x0/0x40 @ 1
[    0.024211] initcall net_defaults_init+0x0/0x40 returned 0 after 0 usecs
[    0.024221] calling  init_default_flow_dissectors+0x0/0x6c @ 1
[    0.024232] initcall init_default_flow_dissectors+0x0/0x6c returned 0 after 0 usecs
[    0.024244] calling  netlink_proto_init+0x0/0x178 @ 1
[    0.024302] NET: Registered protocol family 16
[    0.024331] initcall netlink_proto_init+0x0/0x178 returned 0 after 0 usecs
[    0.024344] calling  genl_init+0x0/0x5c @ 1
[    0.024362] initcall genl_init+0x0/0x5c returned 0 after 0 usecs
[    0.024373] calling  trace_boot_init+0x0/0xf0 @ 1
[    0.024386] initcall trace_boot_init+0x0/0xf0 returned 0 after 0 usecs
[    0.024397] calling  __gnttab_init+0x0/0x48 @ 1
[    0.024408] initcall __gnttab_init+0x0/0x48 returned -19 after 0 usecs
[    0.024484] calling  debug_monitors_init+0x0/0x44 @ 1
[    0.024532] initcall debug_monitors_init+0x0/0x44 returned 0 after 0 usecs
[    0.024545] calling  irq_sysfs_init+0x0/0xc0 @ 1
[    0.024663] initcall irq_sysfs_init+0x0/0xc0 returned 0 after 0 usecs
[    0.024674] calling  dma_atomic_pool_init+0x0/0x154 @ 1
[    0.025243] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations
[    0.025353] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.025526] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.025555] initcall dma_atomic_pool_init+0x0/0x154 returned 0 after 0 usecs
[    0.025568] calling  audit_init+0x0/0x154 @ 1
[    0.025582] audit: initializing netlink subsys (disabled)
[    0.025709] initcall audit_init+0x0/0x154 returned 0 after 0 usecs
[    0.025712] audit: type=2000 audit(0.024:1): state=initialized audit_enabled=0 res=1
[    0.025730] calling  release_early_probes+0x0/0x64 @ 1
[    0.025742] initcall release_early_probes+0x0/0x64 returned 0 after 0 usecs
[    0.025753] calling  bdi_class_init+0x0/0x88 @ 1
[    0.025781] initcall bdi_class_init+0x0/0x88 returned 0 after 0 usecs
[    0.025792] calling  mm_sysfs_init+0x0/0x4c @ 1
[    0.025804] initcall mm_sysfs_init+0x0/0x4c returned 0 after 0 usecs
[    0.025814] calling  init_per_zone_wmark_min+0x0/0x90 @ 1
[    0.025832] initcall init_per_zone_wmark_min+0x0/0x90 returned 0 after 0 usecs
[    0.025845] calling  mpi_init+0x0/0x8c @ 1
[    0.025858] initcall mpi_init+0x0/0x8c returned 0 after 0 usecs
[    0.025868] calling  kobject_uevent_init+0x0/0x24 @ 1
[    0.025896] initcall kobject_uevent_init+0x0/0x24 returned 0 after 0 usecs
[    0.025908] calling  acpi_gpio_setup_params+0x0/0x80 @ 1
[    0.025921] initcall acpi_gpio_setup_params+0x0/0x80 returned 0 after 0 usecs
[    0.025933] calling  pcibus_class_init+0x0/0x34 @ 1
[    0.025953] initcall pcibus_class_init+0x0/0x34 returned 0 after 0 usecs
[    0.025965] calling  pci_driver_init+0x0/0x5c @ 1
[    0.026005] initcall pci_driver_init+0x0/0x5c returned 0 after 0 usecs
[    0.026018] calling  backlight_class_init+0x0/0xcc @ 1
[    0.026033] initcall backlight_class_init+0x0/0xcc returned 0 after 0 usecs
[    0.026044] calling  amba_init+0x0/0x2c @ 1
[    0.026067] initcall amba_init+0x0/0x2c returned 0 after 0 usecs
[    0.026079] calling  xenbus_init+0x0/0x2b0 @ 1
[    0.026091] initcall xenbus_init+0x0/0x2b0 returned -19 after 0 usecs
[    0.026103] calling  tty_class_init+0x0/0x68 @ 1
[    0.026117] initcall tty_class_init+0x0/0x68 returned 0 after 0 usecs
[    0.026127] calling  vtconsole_class_init+0x0/0x104 @ 1
[    0.026171] initcall vtconsole_class_init+0x0/0x104 returned 0 after 0 usecs
[    0.026183] calling  serdev_init+0x0/0x3c @ 1
[    0.026202] initcall serdev_init+0x0/0x3c returned 0 after 0 usecs
[    0.026213] calling  iommu_dev_init+0x0/0x34 @ 1
[    0.026229] initcall iommu_dev_init+0x0/0x34 returned 0 after 0 usecs
[    0.026240] calling  devlink_class_init+0x0/0x70 @ 1
[    0.026256] initcall devlink_class_init+0x0/0x70 returned 0 after 0 usecs
[    0.026267] calling  software_node_init+0x0/0x50 @ 1
[    0.026279] initcall software_node_init+0x0/0x50 returned 0 after 0 usecs
[    0.026290] calling  wakeup_sources_debugfs_init+0x0/0x44 @ 1
[    0.026305] initcall wakeup_sources_debugfs_init+0x0/0x44 returned 0 after 0 usecs
[    0.026318] calling  wakeup_sources_sysfs_init+0x0/0x50 @ 1
[    0.026336] initcall wakeup_sources_sysfs_init+0x0/0x50 returned 0 after 0 usecs
[    0.026348] calling  register_node_type+0x0/0x30 @ 1
[    0.026386] initcall register_node_type+0x0/0x30 returned 0 after 0 usecs
[    0.026398] calling  regmap_initcall+0x0/0x28 @ 1
[    0.026413] initcall regmap_initcall+0x0/0x28 returned 0 after 0 usecs
[    0.026424] calling  sram_init+0x0/0x30 @ 1
[    0.026447] initcall sram_init+0x0/0x30 returned 0 after 0 usecs
[    0.026459] calling  syscon_init+0x0/0x34 @ 1
[    0.026478] initcall syscon_init+0x0/0x34 returned 0 after 0 usecs
[    0.026489] calling  spi_init+0x0/0xd4 @ 1
[    0.026519] initcall spi_init+0x0/0xd4 returned 0 after 0 usecs
[    0.026530] calling  spmi_init+0x0/0x40 @ 1
[    0.026550] initcall spmi_init+0x0/0x40 returned 0 after 0 usecs
[    0.026562] calling  i2c_init+0x0/0x10c @ 1
[    0.026593] initcall i2c_init+0x0/0x10c returned 0 after 0 usecs
[    0.026603] calling  thermal_init+0x0/0x15c @ 1
[    0.026634] thermal_sys: Registered thermal governor 'step_wise'
[    0.026793] initcall thermal_init+0x0/0x15c returned 0 after 0 usecs
[    0.026810] calling  init_menu+0x0/0x2c @ 1
[    0.026909] cpuidle: using governor menu
[    0.026919] initcall init_menu+0x0/0x2c returned 0 after 0 usecs
[    0.026931] calling  pcc_init+0x0/0x74 @ 1
[    0.026941] initcall pcc_init+0x0/0x74 returned -19 after 0 usecs
[    0.026951] calling  rpmsg_init+0x0/0x5c @ 1
[    0.026974] initcall rpmsg_init+0x0/0x5c returned 0 after 0 usecs
[    0.027035] calling  reserve_memblock_reserved_regions+0x0/0x16c @ 1
[    0.027065] initcall reserve_memblock_reserved_regions+0x0/0x16c returned 0 after 0 usecs
[    0.027078] calling  aarch32_alloc_vdso_pages+0x0/0xd8 @ 1
[    0.027094] initcall aarch32_alloc_vdso_pages+0x0/0xd8 returned 0 after 0 usecs
[    0.027105] calling  vdso_init+0x0/0xf8 @ 1
[    0.027116] initcall vdso_init+0x0/0xf8 returned 0 after 0 usecs
[    0.027126] calling  arch_hw_breakpoint_init+0x0/0x100 @ 1
[    0.027138] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.027183] initcall arch_hw_breakpoint_init+0x0/0x100 returned 0 after 0 usecs
[    0.027197] calling  asids_update_limit+0x0/0x120 @ 1
[    0.027209] ASID allocator initialised with 65536 entries
[    0.027216] initcall asids_update_limit+0x0/0x120 returned 0 after 0 usecs
[    0.027226] calling  hugetlbpage_init+0x0/0x44 @ 1
[    0.027240] initcall hugetlbpage_init+0x0/0x44 returned 0 after 0 usecs
[    0.027252] calling  p2m_init+0x0/0x20 @ 1
[    0.027264] initcall p2m_init+0x0/0x20 returned 0 after 0 usecs
[    0.027276] calling  xen_mm_init+0x0/0x9c @ 1
[    0.027286] initcall xen_mm_init+0x0/0x9c returned 0 after 0 usecs
[    0.027297] calling  cryptomgr_init+0x0/0x2c @ 1
[    0.027308] initcall cryptomgr_init+0x0/0x2c returned 0 after 0 usecs
[    0.027318] calling  mpc8xxx_init+0x0/0x30 @ 1
[    0.027342] initcall mpc8xxx_init+0x0/0x30 returned 0 after 0 usecs
[    0.027354] calling  acpi_pci_init+0x0/0x8c @ 1
[    0.027363] initcall acpi_pci_init+0x0/0x8c returned 0 after 0 usecs
[    0.027373] calling  dma_channel_table_init+0x0/0x158 @ 1
[    0.027403] initcall dma_channel_table_init+0x0/0x158 returned 0 after 0 usecs
[    0.027416] calling  dma_bus_init+0x0/0x18c @ 1
[    0.027477] initcall dma_bus_init+0x0/0x18c returned 0 after 0 usecs
[    0.027491] calling  brcmstb_soc_device_init+0x0/0x144 @ 1
[    0.027698] initcall brcmstb_soc_device_init+0x0/0x144 returned 0 after 0 usecs
[    0.027711] calling  register_xen_platform_notifier+0x0/0x60 @ 1
[    0.027723] initcall register_xen_platform_notifier+0x0/0x60 returned 0 after 0 usecs
[    0.027736] calling  register_xen_amba_notifier+0x0/0x64 @ 1
[    0.027748] initcall register_xen_amba_notifier+0x0/0x64 returned 0 after 0 usecs
[    0.027761] calling  register_xen_pci_notifier+0x0/0x54 @ 1
[    0.027772] initcall register_xen_pci_notifier+0x0/0x54 returned 0 after 0 usecs
[    0.027785] calling  pl011_init+0x0/0x60 @ 1
[    0.027794] Serial: AMBA PL011 UART driver
[    0.027816] initcall pl011_init+0x0/0x60 returned 0 after 0 usecs
[    0.027827] calling  cdns_uart_init+0x0/0x30 @ 1
[    0.027844] initcall cdns_uart_init+0x0/0x30 returned 0 after 0 usecs
[    0.027854] calling  iommu_dma_init+0x0/0x28 @ 1
[    0.027867] initcall iommu_dma_init+0x0/0x28 returned 0 after 0 usecs
[    0.027877] calling  dmi_id_init+0x0/0x2ec @ 1
[    0.027889] initcall dmi_id_init+0x0/0x2ec returned -19 after 0 usecs
[    0.027899] calling  of_platform_default_populate_init+0x0/0xdc @ 1
[    0.029744] Machine: Kontron KBox A-230-LS
[    0.029754] SoC family: QorIQ LS1028A
[    0.029760] SoC ID: svr:0x870b0110, Revision: 1.0
[    0.030477] platform 20c0000.spi: Linked as a consumer to 1e00900.clock-controller
[    0.031380] platform 2270000.serial: Linked as a consumer to 22c0000.dma-controller
[    0.031418] platform 2120000.spi: Linked as a consumer to 22c0000.dma-controller
[    0.031853] platform 2000000.i2c: Linked as a sync state only consumer to 2310000.gpio
[    0.034671] platform 3500000.pcie: Linked as a consumer to 5000000.iommu
[    0.034712] platform 3400000.pcie: Linked as a consumer to 5000000.iommu
[    0.034744] platform 3200000.sata: Linked as a consumer to 5000000.iommu
[    0.034776] platform 3110000.usb: Linked as a consumer to 5000000.iommu
[    0.034811] platform 3100000.usb: Linked as a consumer to 5000000.iommu
[    0.034846] platform 22c0000.dma-controller: Linked as a consumer to 5000000.iommu
[    0.034886] platform 2150000.mmc: Linked as a consumer to 5000000.iommu
[    0.034921] platform 2140000.mmc: Linked as a consumer to 5000000.iommu
[    0.035058] platform 8000000.crypto: Linked as a sync state only consumer to 5000000.iommu
[    0.035339] platform 8380000.dma-controller: Linked as a consumer to 5000000.iommu
[    0.035777] platform 1f0000000.pcie: Linked as a consumer to 5000000.iommu
[    0.036163] platform f080000.display: Linked as a consumer to f1f0000.clock-controller
[    0.036201] platform f080000.display: Linked as a consumer to 5000000.iommu
[    0.036438] initcall of_platform_default_populate_init+0x0/0xdc returned 0 after 11718 usecs
[    0.036519] calling  topology_init+0x0/0x130 @ 1
[    0.036651] initcall topology_init+0x0/0x130 returned 0 after 0 usecs
[    0.036664] calling  uid_cache_init+0x0/0xac @ 1
[    0.036684] initcall uid_cache_init+0x0/0xac returned 0 after 0 usecs
[    0.036694] calling  param_sysfs_init+0x0/0x1e8 @ 1
[    0.041587] initcall param_sysfs_init+0x0/0x1e8 returned 0 after 3906 usecs
[    0.041604] calling  user_namespace_sysctl_init+0x0/0x5c @ 1
[    0.041633] initcall user_namespace_sysctl_init+0x0/0x5c returned 0 after 0 usecs
[    0.041645] calling  pm_sysrq_init+0x0/0x34 @ 1
[    0.041778] initcall pm_sysrq_init+0x0/0x34 returned 0 after 0 usecs
[    0.041794] calling  create_proc_profile+0x0/0x118 @ 1
[    0.041807] initcall create_proc_profile+0x0/0x118 returned 0 after 0 usecs
[    0.041817] calling  time_ns_init+0x0/0x18 @ 1
[    0.041829] initcall time_ns_init+0x0/0x18 returned 0 after 0 usecs
[    0.041840] calling  crash_save_vmcoreinfo_init+0x0/0x694 @ 1
[    0.041923] initcall crash_save_vmcoreinfo_init+0x0/0x694 returned 0 after 0 usecs
[    0.041936] calling  crash_notes_memory_init+0x0/0x50 @ 1
[    0.041952] initcall crash_notes_memory_init+0x0/0x50 returned 0 after 0 usecs
[    0.041965] calling  cgroup_sysfs_init+0x0/0x38 @ 1
[    0.041982] initcall cgroup_sysfs_init+0x0/0x38 returned 0 after 0 usecs
[    0.041993] calling  cgroup_namespaces_init+0x0/0x18 @ 1
[    0.042004] initcall cgroup_namespaces_init+0x0/0x18 returned 0 after 0 usecs
[    0.042016] calling  user_namespaces_init+0x0/0x48 @ 1
[    0.042038] initcall user_namespaces_init+0x0/0x48 returned 0 after 0 usecs
[    0.042050] calling  oom_init+0x0/0x4c @ 1
[    0.042116] initcall oom_init+0x0/0x4c returned 0 after 0 usecs
[    0.042130] calling  default_bdi_init+0x0/0xa8 @ 1
[    0.042251] initcall default_bdi_init+0x0/0xa8 returned 0 after 0 usecs
[    0.042264] calling  cgwb_init+0x0/0x4c @ 1
[    0.042276] initcall cgwb_init+0x0/0x4c returned 0 after 0 usecs
[    0.042286] calling  percpu_enable_async+0x0/0x24 @ 1
[    0.042295] initcall percpu_enable_async+0x0/0x24 returned 0 after 0 usecs
[    0.042305] calling  kcompactd_init+0x0/0xcc @ 1
[    0.042364] initcall kcompactd_init+0x0/0xcc returned 0 after 0 usecs
[    0.042378] calling  init_user_reserve+0x0/0x40 @ 1
[    0.042389] initcall init_user_reserve+0x0/0x40 returned 0 after 0 usecs
[    0.042399] calling  init_admin_reserve+0x0/0x40 @ 1
[    0.042409] initcall init_admin_reserve+0x0/0x40 returned 0 after 0 usecs
[    0.042419] calling  init_reserve_notifier+0x0/0x10 @ 1
[    0.042431] initcall init_reserve_notifier+0x0/0x10 returned 0 after 0 usecs
[    0.042443] calling  swap_init_sysfs+0x0/0x84 @ 1
[    0.042458] initcall swap_init_sysfs+0x0/0x84 returned 0 after 0 usecs
[    0.042469] calling  swapfile_init+0x0/0xd0 @ 1
[    0.042479] initcall swapfile_init+0x0/0xd0 returned 0 after 0 usecs
[    0.042489] calling  hugetlb_init+0x0/0x4bc @ 1
[    0.042503] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.042513] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.042521] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.042529] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.042630] initcall hugetlb_init+0x0/0x4bc returned 0 after 0 usecs
[    0.042643] calling  ksm_init+0x0/0x198 @ 1
[    0.042729] initcall ksm_init+0x0/0x198 returned 0 after 0 usecs
[    0.042743] calling  hugepage_init+0x0/0x168 @ 1
[    0.042835] initcall hugepage_init+0x0/0x168 returned 0 after 0 usecs
[    0.042850] calling  mem_cgroup_init+0x0/0x178 @ 1
[    0.042862] initcall mem_cgroup_init+0x0/0x178 returned 0 after 0 usecs
[    0.042873] calling  io_wq_init+0x0/0x54 @ 1
[    0.042884] initcall io_wq_init+0x0/0x54 returned 0 after 0 usecs
[    0.042895] calling  echainiv_module_init+0x0/0x2c @ 1
[    0.042906] initcall echainiv_module_init+0x0/0x2c returned 0 after 0 usecs
[    0.042916] calling  rsa_init+0x0/0x68 @ 1
[    0.042986] initcall rsa_init+0x0/0x68 returned 0 after 0 usecs
[    0.043001] calling  hmac_module_init+0x0/0x2c @ 1
[    0.043011] initcall hmac_module_init+0x0/0x2c returned 0 after 0 usecs
[    0.043021] calling  crypto_null_mod_init+0x0/0x90 @ 1
[    0.043267] initcall crypto_null_mod_init+0x0/0x90 returned 0 after 0 usecs
[    0.043282] calling  md5_mod_init+0x0/0x2c @ 1
[    0.043338] initcall md5_mod_init+0x0/0x2c returned 0 after 0 usecs
[    0.043352] calling  sha1_generic_mod_init+0x0/0x2c @ 1
[    0.043407] initcall sha1_generic_mod_init+0x0/0x2c returned 0 after 0 usecs
[    0.043422] calling  sha256_generic_mod_init+0x0/0x30 @ 1
[    0.043523] initcall sha256_generic_mod_init+0x0/0x30 returned 0 after 0 usecs
[    0.043538] calling  crypto_ecb_module_init+0x0/0x2c @ 1
[    0.043548] initcall crypto_ecb_module_init+0x0/0x2c returned 0 after 0 usecs
[    0.043558] calling  crypto_cbc_module_init+0x0/0x2c @ 1
[    0.043568] initcall crypto_cbc_module_init+0x0/0x2c returned 0 after 0 usecs
[    0.043579] calling  cryptd_init+0x0/0x138 @ 1
[    0.043634] cryptd: max_cpu_qlen set to 1000
[    0.043644] initcall cryptd_init+0x0/0x138 returned 0 after 0 usecs
[    0.043656] calling  aes_init+0x0/0x2c @ 1
[    0.043716] initcall aes_init+0x0/0x2c returned 0 after 0 usecs
[    0.043729] calling  deflate_mod_init+0x0/0x60 @ 1
[    0.043882] initcall deflate_mod_init+0x0/0x60 returned 0 after 0 usecs
[    0.043896] calling  crc32c_mod_init+0x0/0x2c @ 1
[    0.043951] initcall crc32c_mod_init+0x0/0x2c returned 0 after 0 usecs
[    0.043965] calling  crct10dif_mod_init+0x0/0x2c @ 1
[    0.044020] initcall crct10dif_mod_init+0x0/0x2c returned 0 after 0 usecs
[    0.044034] calling  prng_mod_init+0x0/0x30 @ 1
[    0.044104] initcall prng_mod_init+0x0/0x30 returned 0 after 0 usecs
[    0.044117] calling  drbg_init+0x0/0xfc @ 1
[    0.044532] initcall drbg_init+0x0/0xfc returned 0 after 0 usecs
[    0.044548] calling  init_bio+0x0/0xfc @ 1
[    0.044634] initcall init_bio+0x0/0xfc returned 0 after 0 usecs
[    0.044649] calling  blk_settings_init+0x0/0x40 @ 1
[    0.044659] initcall blk_settings_init+0x0/0x40 returned 0 after 0 usecs
[    0.044670] calling  blk_ioc_init+0x0/0x48 @ 1
[    0.044680] initcall blk_ioc_init+0x0/0x48 returned 0 after 0 usecs
[    0.044691] calling  blk_mq_init+0x0/0x104 @ 1
[    0.044702] initcall blk_mq_init+0x0/0x104 returned 0 after 0 usecs
[    0.044713] calling  genhd_device_init+0x0/0x8c @ 1
[    0.044817] initcall genhd_device_init+0x0/0x8c returned 0 after 0 usecs
[    0.044832] calling  blkcg_init+0x0/0x4c @ 1
[    0.044923] initcall blkcg_init+0x0/0x4c returned 0 after 0 usecs
[    0.044937] calling  irq_poll_setup+0x0/0xb0 @ 1
[    0.044949] initcall irq_poll_setup+0x0/0xb0 returned 0 after 0 usecs
[    0.044959] calling  gpiolib_debugfs_init+0x0/0x48 @ 1
[    0.044978] initcall gpiolib_debugfs_init+0x0/0x48 returned 0 after 0 usecs
[    0.044990] calling  max732x_init+0x0/0x30 @ 1
[    0.045015] initcall max732x_init+0x0/0x30 returned 0 after 0 usecs
[    0.045027] calling  pca953x_init+0x0/0x30 @ 1
[    0.045047] initcall pca953x_init+0x0/0x30 returned 0 after 0 usecs
[    0.045059] calling  pwm_debugfs_init+0x0/0x44 @ 1
[    0.045074] initcall pwm_debugfs_init+0x0/0x44 returned 0 after 0 usecs
[    0.045087] calling  pwm_sysfs_init+0x0/0x34 @ 1
[    0.045105] initcall pwm_sysfs_init+0x0/0x34 returned 0 after 0 usecs
[    0.045117] calling  pci_slot_init+0x0/0x68 @ 1
[    0.045133] initcall pci_slot_init+0x0/0x68 returned 0 after 0 usecs
[    0.045144] calling  xgene_pcie_msi_init+0x0/0x30 @ 1
[    0.045240] initcall xgene_pcie_msi_init+0x0/0x30 returned 0 after 0 usecs
[    0.045251] calling  altera_msi_init+0x0/0x30 @ 1
[    0.045295] initcall altera_msi_init+0x0/0x30 returned 0 after 0 usecs
[    0.045305] calling  fbmem_init+0x0/0x108 @ 1
[    0.045354] initcall fbmem_init+0x0/0x108 returned 0 after 0 usecs
[    0.045365] calling  acpi_init+0x0/0x328 @ 1
[    0.045374] ACPI: Interpreter disabled.
[    0.045381] initcall acpi_init+0x0/0x328 returned -19 after 0 usecs
[    0.045391] calling  pnp_init+0x0/0x2c @ 1
[    0.045414] initcall pnp_init+0x0/0x2c returned 0 after 0 usecs
[    0.045425] calling  fsl_edma_init+0x0/0x30 @ 1
[    0.045474] platform 22c0000.dma-controller: probe deferral - supplier 5000000.iommu not ready
[    0.045516] initcall fsl_edma_init+0x0/0x30 returned 0 after 0 usecs
[    0.045528] calling  balloon_init+0x0/0xec @ 1
[    0.045540] initcall balloon_init+0x0/0xec returned -19 after 0 usecs
[    0.045552] calling  xen_setup_shutdown_event+0x0/0x58 @ 1
[    0.045564] initcall xen_setup_shutdown_event+0x0/0x58 returned -19 after 0 usecs
[    0.045576] calling  xenbus_probe_backend_init+0x0/0x64 @ 1
[    0.045600] initcall xenbus_probe_backend_init+0x0/0x64 returned 0 after 0 usecs
[    0.045614] calling  xenbus_probe_frontend_init+0x0/0x4c @ 1
[    0.045640] initcall xenbus_probe_frontend_init+0x0/0x4c returned 0 after 0 usecs
[    0.045654] calling  regulator_fixed_voltage_init+0x0/0x30 @ 1
[    0.045711] initcall regulator_fixed_voltage_init+0x0/0x30 returned 0 after 0 usecs
[    0.045723] calling  gpio_regulator_init+0x0/0x30 @ 1
[    0.045767] initcall gpio_regulator_init+0x0/0x30 returned 0 after 0 usecs
[    0.045777] calling  max8973_init+0x0/0x30 @ 1
[    0.045794] initcall max8973_init+0x0/0x30 returned 0 after 0 usecs
[    0.045805] calling  misc_init+0x0/0xec @ 1
[    0.045823] initcall misc_init+0x0/0xec returned 0 after 0 usecs
[    0.045833] calling  tpm_init+0x0/0x128 @ 1
[    0.045914] initcall tpm_init+0x0/0x128 returned 0 after 0 usecs
[    0.045927] calling  iommu_subsys_init+0x0/0x88 @ 1
[    0.045938] iommu: Default domain type: Translated 
[    0.045945] initcall iommu_subsys_init+0x0/0x88 returned 0 after 0 usecs
[    0.045955] calling  vga_arb_device_init+0x0/0x1c8 @ 1
[    0.046044] vgaarb: loaded
[    0.046051] initcall vga_arb_device_init+0x0/0x1c8 returned 0 after 0 usecs
[    0.046063] calling  register_cpu_capacity_sysctl+0x0/0xa8 @ 1
[    0.046080] initcall register_cpu_capacity_sysctl+0x0/0xa8 returned 0 after 0 usecs
[    0.046091] calling  sec_pmic_init+0x0/0x30 @ 1
[    0.046112] initcall sec_pmic_init+0x0/0x30 returned 0 after 0 usecs
[    0.046124] calling  bd718xx_i2c_init+0x0/0x30 @ 1
[    0.046143] initcall bd718xx_i2c_init+0x0/0x30 returned 0 after 0 usecs
[    0.046154] calling  dma_buf_init+0x0/0xc8 @ 1
[    0.046197] initcall dma_buf_init+0x0/0xc8 returned 0 after 0 usecs
[    0.046210] calling  init_scsi+0x0/0x94 @ 1
[    0.046319] SCSI subsystem initialized
[    0.046326] initcall init_scsi+0x0/0x94 returned 0 after 0 usecs
[    0.046338] calling  ata_init+0x0/0x360 @ 1
[    0.046427] libata version 3.00 loaded.
[    0.046436] initcall ata_init+0x0/0x360 returned 0 after 0 usecs
[    0.046448] calling  pl022_init+0x0/0x2c @ 1
[    0.046475] initcall pl022_init+0x0/0x2c returned 0 after 0 usecs
[    0.046487] calling  phy_init+0x0/0x470 @ 1
[    0.046532] initcall phy_init+0x0/0x470 returned 0 after 0 usecs
[    0.046544] calling  hnae_init+0x0/0x50 @ 1
[    0.046560] initcall hnae_init+0x0/0x50 returned 0 after 0 usecs
[    0.046572] calling  usb_common_init+0x0/0x3c @ 1
[    0.046588] initcall usb_common_init+0x0/0x3c returned 0 after 0 usecs
[    0.046601] calling  ulpi_init+0x0/0x2c @ 1
[    0.046623] initcall ulpi_init+0x0/0x2c returned 0 after 0 usecs
[    0.046635] calling  usb_init+0x0/0x160 @ 1
[    0.046696] usbcore: registered new interface driver usbfs
[    0.046721] usbcore: registered new interface driver hub
[    0.046742] usbcore: registered new device driver usb
[    0.046750] initcall usb_init+0x0/0x160 returned 0 after 0 usecs
[    0.046762] calling  usb_phy_generic_init+0x0/0x30 @ 1
[    0.046827] initcall usb_phy_generic_init+0x0/0x30 returned 0 after 0 usecs
[    0.046840] calling  usb_udc_init+0x0/0x74 @ 1
[    0.046854] initcall usb_udc_init+0x0/0x74 returned 0 after 0 usecs
[    0.046864] calling  typec_init+0x0/0xb0 @ 1
[    0.046896] initcall typec_init+0x0/0xb0 returned 0 after 0 usecs
[    0.046906] calling  usb_roles_init+0x0/0x50 @ 1
[    0.046923] initcall usb_roles_init+0x0/0x50 returned 0 after 0 usecs
[    0.046933] calling  serio_init+0x0/0x54 @ 1
[    0.046952] initcall serio_init+0x0/0x54 returned 0 after 0 usecs
[    0.046961] calling  input_init+0x0/0x130 @ 1
[    0.046982] initcall input_init+0x0/0x130 returned 0 after 0 usecs
[    0.046992] calling  rtc_init+0x0/0x74 @ 1
[    0.047007] initcall rtc_init+0x0/0x74 returned 0 after 0 usecs
[    0.047017] calling  dw_i2c_init_driver+0x0/0x30 @ 1
[    0.047074] initcall dw_i2c_init_driver+0x0/0x30 returned 0 after 0 usecs
[    0.047085] calling  i2c_adap_imx_init+0x0/0x30 @ 1
[    0.047229] imx-i2c 2000000.i2c: can't get pinctrl, bus recovery not supported
[    0.047438] i2c 0-004a: Linked as a sync state only consumer to 2310000.gpio
[    0.047522] i2c i2c-0: IMX I2C adapter registered
[    0.047671] imx-i2c 2030000.i2c: can't get pinctrl, bus recovery not supported
[    0.047781] i2c i2c-1: IMX I2C adapter registered
[    0.047881] imx-i2c 2040000.i2c: can't get pinctrl, bus recovery not supported
[    0.047983] i2c i2c-2: IMX I2C adapter registered
[    0.048061] initcall i2c_adap_imx_init+0x0/0x30 returned 0 after 3906 usecs
[    0.048075] calling  media_devnode_init+0x0/0xa0 @ 1
[    0.048086] mc: Linux media interface: v0.10
[    0.048108] initcall media_devnode_init+0x0/0xa0 returned 0 after 0 usecs
[    0.048119] calling  videodev_init+0x0/0xa8 @ 1
[    0.048129] videodev: Linux video capture interface: v2.00
[    0.048142] initcall videodev_init+0x0/0xa8 returned 0 after 0 usecs
[    0.048152] calling  init_dvbdev+0x0/0x100 @ 1
[    0.048170] initcall init_dvbdev+0x0/0x100 returned 0 after 0 usecs
[    0.048181] calling  pps_init+0x0/0xdc @ 1
[    0.048198] pps_core: LinuxPPS API ver. 1 registered
[    0.048205] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.048214] initcall pps_init+0x0/0xdc returned 0 after 0 usecs
[    0.048225] calling  ptp_init+0x0/0xc0 @ 1
[    0.048239] PTP clock support registered
[    0.048246] initcall ptp_init+0x0/0xc0 returned 0 after 0 usecs
[    0.048256] calling  brcmstb_reboot_init+0x0/0x38 @ 1
[    0.048316] initcall brcmstb_reboot_init+0x0/0x38 returned -19 after 0 usecs
[    0.048327] calling  power_supply_class_init+0x0/0x6c @ 1
[    0.048350] initcall power_supply_class_init+0x0/0x6c returned 0 after 0 usecs
[    0.048361] calling  hwmon_init+0x0/0x58 @ 1
[    0.048376] initcall hwmon_init+0x0/0x58 returned 0 after 0 usecs
[    0.048387] calling  edac_init+0x0/0x98 @ 1
[    0.048396] EDAC MC: Ver: 3.0.0
[    0.048518] initcall edac_init+0x0/0x98 returned 0 after 0 usecs
[    0.048532] calling  psci_idle_init_domains+0x0/0x30 @ 1
[    0.048586] initcall psci_idle_init_domains+0x0/0x30 returned 0 after 0 usecs
[    0.048598] calling  mmc_init+0x0/0x58 @ 1
[    0.048634] initcall mmc_init+0x0/0x58 returned 0 after 0 usecs
[    0.048645] calling  leds_init+0x0/0x78 @ 1
[    0.048661] initcall leds_init+0x0/0x78 returned 0 after 0 usecs
[    0.048671] calling  dmi_init+0x0/0x124 @ 1
[    0.048681] initcall dmi_init+0x0/0x124 returned 0 after 0 usecs
[    0.048692] calling  efisubsys_init+0x0/0x4f4 @ 1
[    0.048703] initcall efisubsys_init+0x0/0x4f4 returned 0 after 0 usecs
[    0.048713] calling  register_gop_device+0x0/0xac @ 1
[    0.048725] initcall register_gop_device+0x0/0xac returned 0 after 0 usecs
[    0.048736] calling  remoteproc_init+0x0/0x4c @ 1
[    0.048764] initcall remoteproc_init+0x0/0x4c returned 0 after 0 usecs
[    0.048775] calling  glink_rpm_init+0x0/0x30 @ 1
[    0.048821] initcall glink_rpm_init+0x0/0x30 returned 0 after 0 usecs
[    0.048832] calling  devfreq_init+0x0/0x10c @ 1
[    0.048912] initcall devfreq_init+0x0/0x10c returned 0 after 0 usecs
[    0.048926] calling  devfreq_event_init+0x0/0x80 @ 1
[    0.048948] initcall devfreq_event_init+0x0/0x80 returned 0 after 0 usecs
[    0.048959] calling  devfreq_simple_ondemand_init+0x0/0x2c @ 1
[    0.048970] initcall devfreq_simple_ondemand_init+0x0/0x2c returned 0 after 0 usecs
[    0.048981] calling  iio_init+0x0/0xb4 @ 1
[    0.049022] initcall iio_init+0x0/0xb4 returned 0 after 0 usecs
[    0.049034] calling  arm_pmu_hp_init+0x0/0x70 @ 1
[    0.049046] initcall arm_pmu_hp_init+0x0/0x70 returned 0 after 0 usecs
[    0.049056] calling  arm_pmu_acpi_init+0x0/0x1f0 @ 1
[    0.049065] initcall arm_pmu_acpi_init+0x0/0x1f0 returned 0 after 0 usecs
[    0.049075] calling  ras_init+0x0/0x28 @ 1
[    0.049093] initcall ras_init+0x0/0x28 returned 0 after 0 usecs
[    0.049104] calling  nvmem_init+0x0/0x30 @ 1
[    0.049124] initcall nvmem_init+0x0/0x30 returned 0 after 0 usecs
[    0.049135] calling  fpga_mgr_class_init+0x0/0x88 @ 1
[    0.049144] FPGA manager framework
[    0.049155] initcall fpga_mgr_class_init+0x0/0x88 returned 0 after 0 usecs
[    0.049166] calling  tee_init+0x0/0xf4 @ 1
[    0.049190] initcall tee_init+0x0/0xf4 returned 0 after 0 usecs
[    0.049200] calling  init_soundcore+0x0/0x6c @ 1
[    0.049215] initcall init_soundcore+0x0/0x6c returned 0 after 0 usecs
[    0.049225] calling  alsa_sound_init+0x0/0xc8 @ 1
[    0.049246] Advanced Linux Sound Architecture Driver Initialized.
[    0.049254] initcall alsa_sound_init+0x0/0xc8 returned 0 after 0 usecs
[    0.049264] calling  proto_init+0x0/0x2c @ 1
[    0.049277] initcall proto_init+0x0/0x2c returned 0 after 0 usecs
[    0.049287] calling  net_dev_init+0x0/0x210 @ 1
[    0.049472] initcall net_dev_init+0x0/0x210 returned 0 after 0 usecs
[    0.049485] calling  neigh_init+0x0/0xb4 @ 1
[    0.049496] initcall neigh_init+0x0/0xb4 returned 0 after 0 usecs
[    0.049506] calling  fib_notifier_init+0x0/0x2c @ 1
[    0.049518] initcall fib_notifier_init+0x0/0x2c returned 0 after 0 usecs
[    0.049529] calling  devlink_init+0x0/0x48 @ 1
[    0.049586] initcall devlink_init+0x0/0x48 returned 0 after 0 usecs
[    0.049598] calling  ethnl_init+0x0/0x84 @ 1
[    0.049627] initcall ethnl_init+0x0/0x84 returned 0 after 0 usecs
[    0.049638] calling  nexthop_init+0x0/0x104 @ 1
[    0.049653] initcall nexthop_init+0x0/0x104 returned 0 after 0 usecs
[    0.049664] calling  watchdog_init+0x0/0xa0 @ 1
[    0.049749] initcall watchdog_init+0x0/0xa0 returned 0 after 0 usecs
[    0.049821] calling  create_debug_debugfs_entry+0x0/0x40 @ 1
[    0.049841] initcall create_debug_debugfs_entry+0x0/0x40 returned 0 after 0 usecs
[    0.049853] calling  iomem_init_inode+0x0/0xbc @ 1
[    0.049891] initcall iomem_init_inode+0x0/0xbc returned 0 after 0 usecs
[    0.049903] calling  clocksource_done_booting+0x0/0x68 @ 1
[    0.049935] clocksource: Switched to clocksource arch_sys_counter
[    0.049944] initcall clocksource_done_booting+0x0/0x68 returned 0 after 15 usecs
[    0.049957] calling  tracer_init_tracefs+0x0/0x2d4 @ 1
[    0.072446] initcall tracer_init_tracefs+0x0/0x2d4 returned 0 after 21948 usecs
[    0.072500] calling  init_trace_printk_function_export+0x0/0x44 @ 1
[    0.072519] initcall init_trace_printk_function_export+0x0/0x44 returned 0 after 5 usecs
[    0.072533] calling  init_graph_tracefs+0x0/0x44 @ 1
[    0.072549] initcall init_graph_tracefs+0x0/0x44 returned 0 after 3 usecs
[    0.072562] calling  init_dynamic_event+0x0/0x54 @ 1
[    0.072576] initcall init_dynamic_event+0x0/0x54 returned 0 after 3 usecs
[    0.072589] calling  init_uprobe_trace+0x0/0x80 @ 1
[    0.072614] initcall init_uprobe_trace+0x0/0x80 returned 0 after 13 usecs
[    0.072627] calling  secretmem_init+0x0/0x40 @ 1
[    0.072675] initcall secretmem_init+0x0/0x40 returned 0 after 34 usecs
[    0.072687] calling  init_pipe_fs+0x0/0x68 @ 1
[    0.072723] initcall init_pipe_fs+0x0/0x68 returned 0 after 22 usecs
[    0.072736] calling  cgroup_writeback_init+0x0/0x4c @ 1
[    0.072751] initcall cgroup_writeback_init+0x0/0x4c returned 0 after 4 usecs
[    0.072762] calling  inotify_user_setup+0x0/0xd8 @ 1
[    0.072788] initcall inotify_user_setup+0x0/0xd8 returned 0 after 14 usecs
[    0.072800] calling  eventpoll_init+0x0/0xf4 @ 1
[    0.072823] initcall eventpoll_init+0x0/0xf4 returned 0 after 12 usecs
[    0.072835] calling  anon_inode_init+0x0/0x78 @ 1
[    0.072862] initcall anon_inode_init+0x0/0x78 returned 0 after 15 usecs
[    0.072875] calling  proc_locks_init+0x0/0x4c @ 1
[    0.072892] initcall proc_locks_init+0x0/0x4c returned 0 after 5 usecs
[    0.072903] calling  iomap_init+0x0/0x38 @ 1
[    0.072940] initcall iomap_init+0x0/0x38 returned 0 after 25 usecs
[    0.072952] calling  dquot_init+0x0/0x13c @ 1
[    0.072963] VFS: Disk quotas dquot_6.6.0
[    0.073000] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.073011] initcall dquot_init+0x0/0x13c returned 0 after 47 usecs
[    0.073023] calling  proc_cmdline_init+0x0/0x44 @ 1
[    0.073037] initcall proc_cmdline_init+0x0/0x44 returned 0 after 2 usecs
[    0.073049] calling  proc_consoles_init+0x0/0x48 @ 1
[    0.073063] initcall proc_consoles_init+0x0/0x48 returned 0 after 3 usecs
[    0.073076] calling  proc_cpuinfo_init+0x0/0x40 @ 1
[    0.073088] initcall proc_cpuinfo_init+0x0/0x40 returned 0 after 1 usecs
[    0.073100] calling  proc_devices_init+0x0/0x48 @ 1
[    0.073112] initcall proc_devices_init+0x0/0x48 returned 0 after 1 usecs
[    0.073124] calling  proc_interrupts_init+0x0/0x48 @ 1
[    0.073137] initcall proc_interrupts_init+0x0/0x48 returned 0 after 1 usecs
[    0.073149] calling  proc_loadavg_init+0x0/0x44 @ 1
[    0.073161] initcall proc_loadavg_init+0x0/0x44 returned 0 after 1 usecs
[    0.073173] calling  proc_meminfo_init+0x0/0x44 @ 1
[    0.073185] initcall proc_meminfo_init+0x0/0x44 returned 0 after 1 usecs
[    0.073197] calling  proc_stat_init+0x0/0x40 @ 1
[    0.073209] initcall proc_stat_init+0x0/0x40 returned 0 after 1 usecs
[    0.073221] calling  proc_uptime_init+0x0/0x44 @ 1
[    0.073233] initcall proc_uptime_init+0x0/0x44 returned 0 after 1 usecs
[    0.073245] calling  proc_version_init+0x0/0x44 @ 1
[    0.073257] initcall proc_version_init+0x0/0x44 returned 0 after 1 usecs
[    0.073269] calling  proc_softirqs_init+0x0/0x44 @ 1
[    0.073281] initcall proc_softirqs_init+0x0/0x44 returned 0 after 1 usecs
[    0.073293] calling  vmcore_init+0x0/0x558 @ 1
[    0.073304] initcall vmcore_init+0x0/0x558 returned 0 after 1 usecs
[    0.073316] calling  proc_kmsg_init+0x0/0x40 @ 1
[    0.073328] initcall proc_kmsg_init+0x0/0x40 returned 0 after 1 usecs
[    0.073340] calling  proc_page_init+0x0/0x7c @ 1
[    0.073354] initcall proc_page_init+0x0/0x7c returned 0 after 3 usecs
[    0.073366] calling  proc_boot_config_init+0x0/0xb0 @ 1
[    0.073380] initcall proc_boot_config_init+0x0/0xb0 returned 0 after 2 usecs
[    0.073392] calling  init_ramfs_fs+0x0/0x2c @ 1
[    0.073404] initcall init_ramfs_fs+0x0/0x2c returned 0 after 1 usecs
[    0.073416] calling  init_hugetlbfs_fs+0x0/0x13c @ 1
[    0.073490] initcall init_hugetlbfs_fs+0x0/0x13c returned 0 after 63 usecs
[    0.073502] calling  blk_scsi_ioctl_init+0x0/0xb0 @ 1
[    0.073513] initcall blk_scsi_ioctl_init+0x0/0xb0 returned 0 after 0 usecs
[    0.073523] calling  acpi_event_init+0x0/0x54 @ 1
[    0.073534] initcall acpi_event_init+0x0/0x54 returned 0 after 0 usecs
[    0.073544] calling  pnp_system_init+0x0/0x2c @ 1
[    0.073584] initcall pnp_system_init+0x0/0x2c returned 0 after 28 usecs
[    0.073596] calling  pnpacpi_init+0x0/0x9c @ 1
[    0.073606] pnp: PnP ACPI: disabled
[    0.073612] initcall pnpacpi_init+0x0/0x9c returned 0 after 5 usecs
[    0.073623] calling  chr_dev_init+0x0/0xd4 @ 1
[    0.080996] initcall chr_dev_init+0x0/0xd4 returned 0 after 7190 usecs
[    0.081018] calling  firmware_class_init+0x0/0xe8 @ 1
[    0.081044] initcall firmware_class_init+0x0/0xe8 returned 0 after 14 usecs
[    0.081056] calling  sysctl_core_init+0x0/0x50 @ 1
[    0.081097] initcall sysctl_core_init+0x0/0x50 returned 0 after 29 usecs
[    0.081108] calling  eth_offload_init+0x0/0x30 @ 1
[    0.081118] initcall eth_offload_init+0x0/0x30 returned 0 after 0 usecs
[    0.081129] calling  ipv4_offload_init+0x0/0x90 @ 1
[    0.081140] initcall ipv4_offload_init+0x0/0x90 returned 0 after 1 usecs
[    0.081151] calling  inet_init+0x0/0x258 @ 1
[    0.081199] NET: Registered protocol family 2
[    0.081504] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[    0.081533] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.081628] TCP bind hash table entries: 32768 (order: 7, 524288 bytes, linear)
[    0.081974] TCP: Hash tables configured (established 32768 bind 32768)
[    0.082076] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.082108] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.082197] initcall inet_init+0x0/0x258 returned 0 after 1010 usecs
[    0.082218] calling  af_unix_init+0x0/0x74 @ 1
[    0.082237] NET: Registered protocol family 1
[    0.082251] initcall af_unix_init+0x0/0x74 returned 0 after 20 usecs
[    0.082264] calling  ipv6_offload_init+0x0/0x94 @ 1
[    0.082277] initcall ipv6_offload_init+0x0/0x94 returned 0 after 1 usecs
[    0.082289] calling  init_sunrpc+0x0/0x8c @ 1
[    0.082608] RPC: Registered named UNIX socket transport module.
[    0.082620] RPC: Registered udp transport module.
[    0.082627] RPC: Registered tcp transport module.
[    0.082634] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.082641] initcall init_sunrpc+0x0/0x8c returned 0 after 333 usecs
[    0.082655] calling  pci_apply_final_quirks+0x0/0x170 @ 1
[    0.082669] PCI: CLS 0 bytes, default 64
[    0.082676] initcall pci_apply_final_quirks+0x0/0x170 returned 0 after 9 usecs
[    0.082686] calling  acpi_reserve_resources+0x0/0x104 @ 1
[    0.082697] initcall acpi_reserve_resources+0x0/0x104 returned 0 after 0 usecs
[    0.082708] calling  populate_rootfs+0x0/0x13c @ 1
[    0.082814] initcall populate_rootfs+0x0/0x13c returned 0 after 92 usecs
[    0.082893] calling  register_arm64_panic_block+0x0/0x38 @ 1
[    0.082905] initcall register_arm64_panic_block+0x0/0x38 returned 0 after 0 usecs
[    0.082917] calling  cpuinfo_regs_init+0x0/0xe0 @ 1
[    0.082984] initcall cpuinfo_regs_init+0x0/0xe0 returned 0 after 55 usecs
[    0.082997] calling  armv8_pmu_driver_init+0x0/0x4c @ 1
[    0.083158] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available
[    0.083338] initcall armv8_pmu_driver_init+0x0/0x4c returned 0 after 321 usecs
[    0.083352] calling  arch_init_uprobes+0x0/0x44 @ 1
[    0.083364] initcall arch_init_uprobes+0x0/0x44 returned 0 after 0 usecs
[    0.083375] calling  arm_init+0x0/0x38 @ 1
[    0.083392] kvm [1]: IPA Size Limit: 44 bits
[    0.083925] kvm [1]: GICv3: no GICV resource entry
[    0.083935] kvm [1]: disabling GICv2 emulation
[    0.083954] kvm [1]: GIC system register CPU interface enabled
[    0.083993] kvm [1]: vgic interrupt IRQ9
[    0.084053] kvm [1]: Hyp mode initialized successfully
[    0.084201] initcall arm_init+0x0/0x38 returned 0 after 794 usecs
[    0.084216] calling  cpu_feature_match_SHA1_init+0x0/0x4c @ 1
[    0.084229] initcall cpu_feature_match_SHA1_init+0x0/0x4c returned -19 after 0 usecs
[    0.084242] calling  cpu_feature_match_SHA2_init+0x0/0x50 @ 1
[    0.084253] initcall cpu_feature_match_SHA2_init+0x0/0x50 returned -19 after 0 usecs
[    0.084266] calling  ghash_ce_mod_init+0x0/0x60 @ 1
[    0.084375] initcall ghash_ce_mod_init+0x0/0x60 returned 0 after 94 usecs
[    0.084389] calling  cpu_feature_match_AES_init+0x0/0x4c @ 1
[    0.084400] initcall cpu_feature_match_AES_init+0x0/0x4c returned -19 after 0 usecs
[    0.084413] calling  aes_mod_init+0x0/0x44 @ 1
[    0.084424] initcall aes_mod_init+0x0/0x44 returned -19 after 0 usecs
[    0.084436] calling  cpu_feature_match_AES_init+0x0/0xd8 @ 1
[    0.084447] initcall cpu_feature_match_AES_init+0x0/0xd8 returned -19 after 0 usecs
[    0.084459] calling  sha256_mod_init+0x0/0x84 @ 1
[    0.093974] initcall sha256_mod_init+0x0/0x84 returned 0 after 9281 usecs
[    0.093994] calling  aes_init+0x0/0x2c @ 1
[    0.094062] initcall aes_init+0x0/0x2c returned 0 after 56 usecs
[    0.094077] calling  proc_execdomains_init+0x0/0x44 @ 1
[    0.094093] initcall proc_execdomains_init+0x0/0x44 returned 0 after 4 usecs
[    0.094105] calling  register_warn_debugfs+0x0/0x44 @ 1
[    0.094125] initcall register_warn_debugfs+0x0/0x44 returned 0 after 7 usecs
[    0.094138] calling  cpuhp_sysfs_init+0x0/0xc8 @ 1
[    0.094170] initcall cpuhp_sysfs_init+0x0/0xc8 returned 0 after 20 usecs
[    0.094183] calling  ioresources_init+0x0/0x7c @ 1
[    0.094196] initcall ioresources_init+0x0/0x7c returned 0 after 4 usecs
[    0.094206] calling  snapshot_device_init+0x0/0x2c @ 1
[    0.094273] initcall snapshot_device_init+0x0/0x2c returned 0 after 56 usecs
[    0.094285] calling  irq_pm_init_ops+0x0/0x30 @ 1
[    0.094295] initcall irq_pm_init_ops+0x0/0x30 returned 0 after 0 usecs
[    0.094306] calling  irq_debugfs_init+0x0/0xa8 @ 1
[    0.094744] initcall irq_debugfs_init+0x0/0xa8 returned 0 after 418 usecs
[    0.094757] calling  timekeeping_init_ops+0x0/0x34 @ 1
[    0.094769] initcall timekeeping_init_ops+0x0/0x34 returned 0 after 0 usecs
[    0.094779] calling  init_clocksource_sysfs+0x0/0x48 @ 1
[    0.094846] initcall init_clocksource_sysfs+0x0/0x48 returned 0 after 54 usecs
[    0.094860] calling  init_timer_list_procfs+0x0/0x50 @ 1
[    0.094873] initcall init_timer_list_procfs+0x0/0x50 returned 0 after 2 usecs
[    0.094884] calling  alarmtimer_init+0x0/0xc8 @ 1
[    0.094945] initcall alarmtimer_init+0x0/0xc8 returned 0 after 49 usecs
[    0.094957] calling  init_posix_timers+0x0/0x48 @ 1
[    0.094975] initcall init_posix_timers+0x0/0x48 returned 0 after 6 usecs
[    0.094986] calling  clockevents_init_sysfs+0x0/0xfc @ 1
[    0.095073] initcall clockevents_init_sysfs+0x0/0xfc returned 0 after 75 usecs
[    0.095088] calling  sched_clock_syscore_init+0x0/0x34 @ 1
[    0.095098] initcall sched_clock_syscore_init+0x0/0x34 returned 0 after 0 usecs
[    0.095110] calling  proc_modules_init+0x0/0x44 @ 1
[    0.095123] initcall proc_modules_init+0x0/0x44 returned 0 after 2 usecs
[    0.095135] calling  kallsyms_init+0x0/0x40 @ 1
[    0.095147] initcall kallsyms_init+0x0/0x40 returned 0 after 1 usecs
[    0.095158] calling  pid_namespaces_init+0x0/0x48 @ 1
[    0.095185] initcall pid_namespaces_init+0x0/0x48 returned 0 after 15 usecs
[    0.095197] calling  ikconfig_init+0x0/0x64 @ 1
[    0.095210] initcall ikconfig_init+0x0/0x64 returned 0 after 2 usecs
[    0.095222] calling  audit_watch_init+0x0/0x54 @ 1
[    0.095233] initcall audit_watch_init+0x0/0x54 returned 0 after 0 usecs
[    0.095244] calling  audit_fsnotify_init+0x0/0x54 @ 1
[    0.095257] initcall audit_fsnotify_init+0x0/0x54 returned 0 after 2 usecs
[    0.095269] calling  audit_tree_init+0x0/0x9c @ 1
[    0.095282] initcall audit_tree_init+0x0/0x9c returned 0 after 2 usecs
[    0.095294] calling  seccomp_sysctl_init+0x0/0x44 @ 1
[    0.095314] initcall seccomp_sysctl_init+0x0/0x44 returned 0 after 9 usecs
[    0.095326] calling  utsname_sysctl_init+0x0/0x30 @ 1
[    0.095347] initcall utsname_sysctl_init+0x0/0x30 returned 0 after 9 usecs
[    0.095358] calling  init_tracepoints+0x0/0x50 @ 1
[    0.095371] initcall init_tracepoints+0x0/0x50 returned 0 after 1 usecs
[    0.095382] calling  perf_event_sysfs_init+0x0/0xb8 @ 1
[    0.095535] initcall perf_event_sysfs_init+0x0/0xb8 returned 0 after 137 usecs
[    0.095550] calling  system_trusted_keyring_init+0x0/0x78 @ 1
[    0.095559] Initialise system trusted keyrings
[    0.095573] initcall system_trusted_keyring_init+0x0/0x78 returned 0 after 12 usecs
[    0.095584] calling  kswapd_init+0x0/0x90 @ 1
[    0.095674] initcall kswapd_init+0x0/0x90 returned 0 after 79 usecs
[    0.095687] calling  extfrag_debug_init+0x0/0x80 @ 1
[    0.095709] initcall extfrag_debug_init+0x0/0x80 returned 0 after 12 usecs
[    0.095719] calling  mm_compute_batch_init+0x0/0x30 @ 1
[    0.095729] initcall mm_compute_batch_init+0x0/0x30 returned 0 after 0 usecs
[    0.095739] calling  slab_proc_init+0x0/0x44 @ 1
[    0.095751] initcall slab_proc_init+0x0/0x44 returned 0 after 3 usecs
[    0.095762] calling  workingset_init+0x0/0xc4 @ 1
[    0.095771] workingset: timestamp_bits=44 max_order=20 bucket_order=0
[    0.095784] initcall workingset_init+0x0/0xc4 returned 0 after 13 usecs
[    0.095795] calling  proc_vmalloc_init+0x0/0x50 @ 1
[    0.095806] initcall proc_vmalloc_init+0x0/0x50 returned 0 after 1 usecs
[    0.095816] calling  memblock_init_debugfs+0x0/0x90 @ 1
[    0.095836] initcall memblock_init_debugfs+0x0/0x90 returned 0 after 9 usecs
[    0.095847] calling  procswaps_init+0x0/0x44 @ 1
[    0.095859] initcall procswaps_init+0x0/0x44 returned 0 after 2 usecs
[    0.095870] calling  slab_sysfs_init+0x0/0x128 @ 1
[    0.098572] initcall slab_sysfs_init+0x0/0x128 returned 0 after 2628 usecs
[    0.098593] calling  fcntl_init+0x0/0x48 @ 1
[    0.098610] initcall fcntl_init+0x0/0x48 returned 0 after 5 usecs
[    0.098621] calling  proc_filesystems_init+0x0/0x44 @ 1
[    0.098637] initcall proc_filesystems_init+0x0/0x44 returned 0 after 5 usecs
[    0.098649] calling  start_dirtytime_writeback+0x0/0x4c @ 1
[    0.098664] initcall start_dirtytime_writeback+0x0/0x4c returned 0 after 3 usecs
[    0.098676] calling  blkdev_init+0x0/0x38 @ 1
[    0.098699] initcall blkdev_init+0x0/0x38 returned 0 after 12 usecs
[    0.098711] calling  dio_init+0x0/0x48 @ 1
[    0.098771] initcall dio_init+0x0/0x48 returned 0 after 49 usecs
[    0.098784] calling  dnotify_init+0x0/0x98 @ 1
[    0.098837] initcall dnotify_init+0x0/0x98 returned 0 after 41 usecs
[    0.098850] calling  fanotify_user_setup+0x0/0xb4 @ 1
[    0.098917] initcall fanotify_user_setup+0x0/0xb4 returned 0 after 54 usecs
[    0.098930] calling  aio_setup+0x0/0xa0 @ 1
[    0.099022] initcall aio_setup+0x0/0xa0 returned 0 after 79 usecs
[    0.099036] calling  io_uring_init+0x0/0x4c @ 1
[    0.099053] initcall io_uring_init+0x0/0x4c returned 0 after 5 usecs
[    0.099065] calling  mbcache_init+0x0/0x54 @ 1
[    0.099115] initcall mbcache_init+0x0/0x54 returned 0 after 38 usecs
[    0.099128] calling  init_grace+0x0/0x2c @ 1
[    0.099141] initcall init_grace+0x0/0x2c returned 0 after 2 usecs
[    0.099152] calling  init_devpts_fs+0x0/0x4c @ 1
[    0.099179] initcall init_devpts_fs+0x0/0x4c returned 0 after 15 usecs
[    0.099191] calling  ext4_init_fs+0x0/0x1a8 @ 1
[    0.099510] initcall ext4_init_fs+0x0/0x1a8 returned 0 after 300 usecs
[    0.099524] calling  init_ext2_fs+0x0/0x88 @ 1
[    0.099568] initcall init_ext2_fs+0x0/0x88 returned 0 after 32 usecs
[    0.099581] calling  journal_init+0x0/0x13c @ 1
[    0.099677] initcall journal_init+0x0/0x13c returned 0 after 82 usecs
[    0.099691] calling  init_squashfs_fs+0x0/0x94 @ 1
[    0.099736] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.099744] initcall init_squashfs_fs+0x0/0x94 returned 0 after 40 usecs
[    0.099756] calling  init_fat_fs+0x0/0x6c @ 1
[    0.099827] initcall init_fat_fs+0x0/0x6c returned 0 after 61 usecs
[    0.099838] calling  init_vfat_fs+0x0/0x2c @ 1
[    0.099847] initcall init_vfat_fs+0x0/0x2c returned 0 after 1 usecs
[    0.099857] calling  init_nfs_fs+0x0/0x168 @ 1
[    0.100196] initcall init_nfs_fs+0x0/0x168 returned 0 after 322 usecs
[    0.100210] calling  init_nfs_v2+0x0/0x30 @ 1
[    0.100219] initcall init_nfs_v2+0x0/0x30 returned 0 after 0 usecs
[    0.100229] calling  init_nfs_v3+0x0/0x30 @ 1
[    0.100237] initcall init_nfs_v3+0x0/0x30 returned 0 after 0 usecs
[    0.100247] calling  init_nfs_v4+0x0/0x6c @ 1
[    0.100255] NFS: Registering the id_resolver key type
[    0.100273] Key type id_resolver registered
[    0.100280] Key type id_legacy registered
[    0.100332] initcall init_nfs_v4+0x0/0x6c returned 0 after 74 usecs
[    0.100344] calling  nfs4filelayout_init+0x0/0x44 @ 1
[    0.100353] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.100361] initcall nfs4filelayout_init+0x0/0x44 returned 0 after 8 usecs
[    0.100371] calling  init_nlm+0x0/0x8c @ 1
[    0.100389] initcall init_nlm+0x0/0x8c returned 0 after 10 usecs
[    0.100400] calling  init_nls_cp437+0x0/0x30 @ 1
[    0.100409] initcall init_nls_cp437+0x0/0x30 returned 0 after 0 usecs
[    0.100419] calling  init_nls_iso8859_1+0x0/0x30 @ 1
[    0.100428] initcall init_nls_iso8859_1+0x0/0x30 returned 0 after 0 usecs
[    0.100437] calling  init_autofs_fs+0x0/0x48 @ 1
[    0.100528] initcall init_autofs_fs+0x0/0x48 returned 0 after 80 usecs
[    0.100540] calling  init_v9fs+0x0/0x10c @ 1
[    0.100549] 9p: Installing v9fs 9p2000 file system support
[    0.100594] initcall init_v9fs+0x0/0x10c returned 0 after 44 usecs
[    0.100606] calling  efivarfs_init+0x0/0x3c @ 1
[    0.100615] initcall efivarfs_init+0x0/0x3c returned -19 after 0 usecs
[    0.100625] calling  ipc_init+0x0/0x40 @ 1
[    0.100646] initcall ipc_init+0x0/0x40 returned 0 after 12 usecs
[    0.100657] calling  ipc_sysctl_init+0x0/0x30 @ 1
[    0.100679] initcall ipc_sysctl_init+0x0/0x30 returned 0 after 12 usecs
[    0.100690] calling  init_mqueue_fs+0x0/0x104 @ 1
[    0.100772] initcall init_mqueue_fs+0x0/0x104 returned 0 after 71 usecs
[    0.100785] calling  key_proc_init+0x0/0x94 @ 1
[    0.100798] initcall key_proc_init+0x0/0x94 returned 0 after 4 usecs
[    0.100808] calling  crypto_algapi_init+0x0/0x28 @ 1
[    0.100820] initcall crypto_algapi_init+0x0/0x28 returned 0 after 2 usecs
[    0.100830] calling  jent_mod_init+0x0/0x4c @ 1
[    0.130699] initcall jent_mod_init+0x0/0x4c returned 0 after 29159 usecs
[    0.130714] calling  asymmetric_key_init+0x0/0x2c @ 1
[    0.130725] Key type asymmetric registered
[    0.130731] initcall asymmetric_key_init+0x0/0x2c returned 0 after 7 usecs
[    0.130742] calling  x509_key_init+0x0/0x2c @ 1
[    0.130751] Asymmetric key parser 'x509' registered
[    0.130758] initcall x509_key_init+0x0/0x2c returned 0 after 7 usecs
[    0.130768] calling  proc_genhd_init+0x0/0x74 @ 1
[    0.130782] initcall proc_genhd_init+0x0/0x74 returned 0 after 4 usecs
[    0.130793] calling  bsg_init+0x0/0x134 @ 1
[    0.130816] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    0.130825] initcall bsg_init+0x0/0x134 returned 0 after 22 usecs
[    0.130835] calling  deadline_init+0x0/0x2c @ 1
[    0.130846] io scheduler mq-deadline registered
[    0.130853] initcall deadline_init+0x0/0x2c returned 0 after 7 usecs
[    0.130863] calling  kyber_init+0x0/0x2c @ 1
[    0.130873] io scheduler kyber registered
[    0.130879] initcall kyber_init+0x0/0x2c returned 0 after 7 usecs
[    0.130890] calling  crc_t10dif_mod_init+0x0/0x5c @ 1
[    0.130911] initcall crc_t10dif_mod_init+0x0/0x5c returned 0 after 11 usecs
[    0.130922] calling  percpu_counter_startup+0x0/0x74 @ 1
[    0.130957] initcall percpu_counter_startup+0x0/0x74 returned 0 after 23 usecs
[    0.130977] calling  audit_classes_init+0x0/0xb8 @ 1
[    0.130996] initcall audit_classes_init+0x0/0xb8 returned 0 after 7 usecs
[    0.131008] calling  sg_pool_init+0x0/0xf4 @ 1
[    0.131105] initcall sg_pool_init+0x0/0xf4 returned 0 after 84 usecs
[    0.131118] calling  ls_scfg_msi_driver_init+0x0/0x34 @ 1
[    0.131259] initcall ls_scfg_msi_driver_init+0x0/0x34 returned 0 after 125 usecs
[    0.131273] calling  sl28cpld_intc_driver_init+0x0/0x30 @ 1
[    0.131322] initcall sl28cpld_intc_driver_init+0x0/0x30 returned 0 after 35 usecs
[    0.131336] calling  brcm_gisb_driver_init+0x0/0x38 @ 1
[    0.131415] initcall brcm_gisb_driver_init+0x0/0x38 returned -19 after 65 usecs
[    0.131429] calling  simple_pm_bus_driver_init+0x0/0x30 @ 1
[    0.131478] initcall simple_pm_bus_driver_init+0x0/0x30 returned 0 after 36 usecs
[    0.131493] calling  vexpress_syscfg_driver_init+0x0/0x30 @ 1
[    0.131529] initcall vexpress_syscfg_driver_init+0x0/0x30 returned 0 after 24 usecs
[    0.131544] calling  phy_core_init+0x0/0x78 @ 1
[    0.131562] initcall phy_core_init+0x0/0x78 returned 0 after 6 usecs
[    0.131574] calling  xgene_phy_driver_init+0x0/0x30 @ 1
[    0.131621] initcall xgene_phy_driver_init+0x0/0x30 returned 0 after 35 usecs
[    0.131634] calling  imx8mq_usb_phy_driver_init+0x0/0x30 @ 1
[    0.131690] initcall imx8mq_usb_phy_driver_init+0x0/0x30 returned 0 after 42 usecs
[    0.131704] calling  qcom_usb_hs_phy_driver_init+0x0/0x30 @ 1
[    0.131725] initcall qcom_usb_hs_phy_driver_init+0x0/0x30 returned 0 after 9 usecs
[    0.131738] calling  samsung_usb2_phy_driver_init+0x0/0x30 @ 1
[    0.131777] initcall samsung_usb2_phy_driver_init+0x0/0x30 returned 0 after 26 usecs
[    0.131791] calling  max77620_pinctrl_driver_init+0x0/0x30 @ 1
[    0.131825] initcall max77620_pinctrl_driver_init+0x0/0x30 returned 0 after 21 usecs
[    0.131839] calling  pcs_driver_init+0x0/0x30 @ 1
[    0.131915] initcall pcs_driver_init+0x0/0x30 returned 0 after 63 usecs
[    0.131927] calling  bgpio_driver_init+0x0/0x30 @ 1
[    0.131987] initcall bgpio_driver_init+0x0/0x30 returned 0 after 47 usecs
[    0.132000] calling  dwapb_gpio_driver_init+0x0/0x30 @ 1
[    0.132055] initcall dwapb_gpio_driver_init+0x0/0x30 returned 0 after 42 usecs
[    0.132068] calling  max77620_gpio_driver_init+0x0/0x30 @ 1
[    0.132102] initcall max77620_gpio_driver_init+0x0/0x30 returned 0 after 21 usecs
[    0.132116] calling  mb86s70_gpio_driver_init+0x0/0x30 @ 1
[    0.132163] initcall mb86s70_gpio_driver_init+0x0/0x30 returned 0 after 35 usecs
[    0.132177] calling  pl061_gpio_driver_init+0x0/0x2c @ 1
[    0.132200] initcall pl061_gpio_driver_init+0x0/0x2c returned 0 after 11 usecs
[    0.132214] calling  sl28cpld_gpio_driver_init+0x0/0x30 @ 1
[    0.132271] initcall sl28cpld_gpio_driver_init+0x0/0x30 returned 0 after 44 usecs
[    0.132285] calling  xgene_gpio_driver_init+0x0/0x30 @ 1
[    0.132332] initcall xgene_gpio_driver_init+0x0/0x30 returned 0 after 34 usecs
[    0.132345] calling  sl28cpld_pwm_driver_init+0x0/0x30 @ 1
[    0.132392] initcall sl28cpld_pwm_driver_init+0x0/0x30 returned 0 after 34 usecs
[    0.132406] calling  pcie_portdrv_init+0x0/0x60 @ 1
[    0.132471] initcall pcie_portdrv_init+0x0/0x60 returned 0 after 52 usecs
[    0.132484] calling  pci_proc_init+0x0/0xa0 @ 1
[    0.132498] initcall pci_proc_init+0x0/0xa0 returned 0 after 5 usecs
[    0.132508] calling  pci_hotplug_init+0x0/0x18 @ 1
[    0.132518] initcall pci_hotplug_init+0x0/0x18 returned 0 after 0 usecs
[    0.132527] calling  gen_pci_driver_init+0x0/0x30 @ 1
[    0.132588] platform 1f0000000.pcie: probe deferral - supplier 5000000.iommu not ready
[    0.132614] initcall gen_pci_driver_init+0x0/0x30 returned 0 after 76 usecs
[    0.132625] calling  altera_pcie_driver_init+0x0/0x30 @ 1
[    0.132676] initcall altera_pcie_driver_init+0x0/0x30 returned 0 after 40 usecs
[    0.132687] calling  ls_pcie_driver_init+0x0/0x38 @ 1
[    0.132762] platform 3400000.pcie: probe deferral - supplier 5000000.iommu not ready
[    0.132777] platform 3500000.pcie: probe deferral - supplier 5000000.iommu not ready
[    0.132818] initcall ls_pcie_driver_init+0x0/0x38 returned -19 after 119 usecs
[    0.132830] calling  kirin_pcie_driver_init+0x0/0x34 @ 1
[    0.132874] initcall kirin_pcie_driver_init+0x0/0x34 returned 0 after 33 usecs
[    0.132885] calling  hisi_pcie_almost_ecam_driver_init+0x0/0x30 @ 1
[    0.132935] initcall hisi_pcie_almost_ecam_driver_init+0x0/0x30 returned 0 after 38 usecs
[    0.132947] calling  thunder_ecam_driver_init+0x0/0x30 @ 1
[    0.132991] initcall thunder_ecam_driver_init+0x0/0x30 returned 0 after 33 usecs
[    0.133002] calling  thunder_pem_driver_init+0x0/0x30 @ 1
[    0.133047] initcall thunder_pem_driver_init+0x0/0x30 returned 0 after 35 usecs
[    0.133059] calling  xgene_pcie_driver_init+0x0/0x30 @ 1
[    0.133102] initcall xgene_pcie_driver_init+0x0/0x30 returned 0 after 33 usecs
[    0.133113] calling  xenfb_init+0x0/0x5c @ 1
[    0.133122] initcall xenfb_init+0x0/0x5c returned -19 after 0 usecs
[    0.133132] calling  efifb_driver_init+0x0/0x30 @ 1
[    0.133162] initcall efifb_driver_init+0x0/0x30 returned 0 after 21 usecs
[    0.133173] calling  ged_driver_init+0x0/0x30 @ 1
[    0.133208] initcall ged_driver_init+0x0/0x30 returned 0 after 24 usecs
[    0.133219] calling  acpi_ac_init+0x0/0xd0 @ 1
[    0.133229] initcall acpi_ac_init+0x0/0xd0 returned -19 after 0 usecs
[    0.133239] calling  acpi_button_driver_init+0x0/0x84 @ 1
[    0.133250] initcall acpi_button_driver_init+0x0/0x84 returned 0 after 0 usecs
[    0.133261] calling  acpi_fan_driver_init+0x0/0x30 @ 1
[    0.133293] initcall acpi_fan_driver_init+0x0/0x30 returned 0 after 21 usecs
[    0.133305] calling  acpi_processor_driver_init+0x0/0xec @ 1
[    0.133315] initcall acpi_processor_driver_init+0x0/0xec returned 0 after 0 usecs
[    0.133327] calling  acpi_thermal_init+0x0/0x9c @ 1
[    0.133979] initcall acpi_thermal_init+0x0/0x9c returned -19 after 627 usecs
[    0.133995] calling  acpi_battery_init+0x0/0x5c @ 1
[    0.134006] initcall acpi_battery_init+0x0/0x5c returned -19 after 0 usecs
[    0.134017] calling  acpi_hed_driver_init+0x0/0x30 @ 1
[    0.134027] initcall acpi_hed_driver_init+0x0/0x30 returned -19 after 0 usecs
[    0.134038] calling  erst_init+0x0/0x2fc @ 1
[    0.134048] initcall erst_init+0x0/0x2fc returned 0 after 0 usecs
[    0.134058] calling  ghes_init+0x0/0x118 @ 1
[    0.134068] initcall ghes_init+0x0/0x118 returned -19 after 0 usecs
[    0.134078] calling  einj_init+0x0/0x4a8 @ 1
[    0.134088] EINJ: ACPI disabled.
[    0.134094] initcall einj_init+0x0/0x4a8 returned -19 after 5 usecs
[    0.134105] calling  gtdt_sbsa_gwdt_init+0x0/0x120 @ 1
[    0.134116] initcall gtdt_sbsa_gwdt_init+0x0/0x120 returned 0 after 0 usecs
[    0.134127] calling  of_fixed_factor_clk_driver_init+0x0/0x30 @ 1
[    0.134184] initcall of_fixed_factor_clk_driver_init+0x0/0x30 returned 0 after 44 usecs
[    0.134198] calling  of_fixed_clk_driver_init+0x0/0x30 @ 1
[    0.134247] initcall of_fixed_clk_driver_init+0x0/0x30 returned 0 after 37 usecs
[    0.134260] calling  gpio_clk_driver_init+0x0/0x30 @ 1
[    0.134314] initcall gpio_clk_driver_init+0x0/0x30 returned 0 after 41 usecs
[    0.134326] calling  cs2000_driver_init+0x0/0x30 @ 1
[    0.134362] initcall cs2000_driver_init+0x0/0x30 returned 0 after 24 usecs
[    0.134374] calling  fsl_flexspi_clk_driver_init+0x0/0x30 @ 1
[    0.134534] initcall fsl_flexspi_clk_driver_init+0x0/0x30 returned 0 after 145 usecs
[    0.134550] calling  fsl_sai_clk_driver_init+0x0/0x30 @ 1
[    0.134600] initcall fsl_sai_clk_driver_init+0x0/0x30 returned 0 after 38 usecs
[    0.134614] calling  plldig_clk_driver_init+0x0/0x30 @ 1
[    0.134716] initcall plldig_clk_driver_init+0x0/0x30 returned 0 after 88 usecs
[    0.134731] calling  clk_pwm_driver_init+0x0/0x30 @ 1
[    0.134779] initcall clk_pwm_driver_init+0x0/0x30 returned 0 after 36 usecs
[    0.134792] calling  clockgen_cpufreq_init+0x0/0xac @ 1
[    0.134858] initcall clockgen_cpufreq_init+0x0/0xac returned 0 after 54 usecs
[    0.134872] calling  rk808_clkout_driver_init+0x0/0x30 @ 1
[    0.134910] initcall rk808_clkout_driver_init+0x0/0x30 returned 0 after 26 usecs
[    0.134924] calling  s2mps11_clk_driver_init+0x0/0x30 @ 1
[    0.134960] initcall s2mps11_clk_driver_init+0x0/0x30 returned 0 after 24 usecs
[    0.134973] calling  scpi_clocks_driver_init+0x0/0x30 @ 1
[    0.135021] initcall scpi_clocks_driver_init+0x0/0x30 returned 0 after 35 usecs
[    0.135035] calling  mv_xor_v2_driver_init+0x0/0x30 @ 1
[    0.135082] initcall mv_xor_v2_driver_init+0x0/0x30 returned 0 after 35 usecs
[    0.135095] calling  pl330_driver_init+0x0/0x2c @ 1
[    0.135118] initcall pl330_driver_init+0x0/0x2c returned 0 after 11 usecs
[    0.135130] calling  hidma_mgmt_init+0x0/0x1f0 @ 1
[    0.135233] initcall hidma_mgmt_init+0x0/0x1f0 returned 0 after 89 usecs
[    0.135245] calling  hidma_driver_init+0x0/0x30 @ 1
[    0.135304] initcall hidma_driver_init+0x0/0x30 returned 0 after 47 usecs
[    0.135317] calling  dpaa2_console_driver_init+0x0/0x30 @ 1
[    0.135367] initcall dpaa2_console_driver_init+0x0/0x30 returned 0 after 37 usecs
[    0.135380] calling  virtio_mmio_init+0x0/0x30 @ 1
[    0.135430] initcall virtio_mmio_init+0x0/0x30 returned 0 after 37 usecs
[    0.135443] calling  virtio_pci_driver_init+0x0/0x38 @ 1
[    0.135473] initcall virtio_pci_driver_init+0x0/0x38 returned 0 after 18 usecs
[    0.135486] calling  virtio_balloon_driver_init+0x0/0x2c @ 1
[    0.135507] initcall virtio_balloon_driver_init+0x0/0x2c returned 0 after 8 usecs
[    0.135520] calling  xenbus_probe_initcall+0x0/0x44 @ 1
[    0.135532] initcall xenbus_probe_initcall+0x0/0x44 returned 0 after 0 usecs
[    0.135544] calling  xenbus_init+0x0/0x60 @ 1
[    0.135554] initcall xenbus_init+0x0/0x60 returned -19 after 0 usecs
[    0.135566] calling  xenbus_backend_init+0x0/0x6c @ 1
[    0.135577] initcall xenbus_backend_init+0x0/0x6c returned -19 after 0 usecs
[    0.135589] calling  evtchn_init+0x0/0x70 @ 1
[    0.135600] initcall evtchn_init+0x0/0x70 returned -19 after 0 usecs
[    0.135612] calling  gntdev_init+0x0/0x78 @ 1
[    0.135623] initcall gntdev_init+0x0/0x78 returned -19 after 0 usecs
[    0.135634] calling  gntalloc_init+0x0/0x60 @ 1
[    0.135645] initcall gntalloc_init+0x0/0x60 returned -19 after 0 usecs
[    0.135657] calling  xenfs_init+0x0/0x40 @ 1
[    0.135668] initcall xenfs_init+0x0/0x40 returned 0 after 0 usecs
[    0.135679] calling  hyper_sysfs_init+0x0/0x134 @ 1
[    0.135691] initcall hyper_sysfs_init+0x0/0x134 returned -19 after 0 usecs
[    0.135702] calling  hypervisor_subsys_init+0x0/0x40 @ 1
[    0.135714] initcall hypervisor_subsys_init+0x0/0x40 returned -19 after 0 usecs
[    0.135727] calling  privcmd_init+0x0/0x90 @ 1
[    0.135738] initcall privcmd_init+0x0/0x90 returned -19 after 0 usecs
[    0.135750] calling  axp20x_regulator_driver_init+0x0/0x30 @ 1
[    0.135783] initcall axp20x_regulator_driver_init+0x0/0x30 returned 0 after 23 usecs
[    0.135795] calling  bd718xx_regulator_init+0x0/0x30 @ 1
[    0.135828] initcall bd718xx_regulator_init+0x0/0x30 returned 0 after 23 usecs
[    0.135839] calling  bd9571mwv_regulator_driver_init+0x0/0x30 @ 1
[    0.135872] initcall bd9571mwv_regulator_driver_init+0x0/0x30 returned 0 after 22 usecs
[    0.135884] calling  fan53555_regulator_driver_init+0x0/0x30 @ 1
[    0.135914] initcall fan53555_regulator_driver_init+0x0/0x30 returned 0 after 20 usecs
[    0.135926] calling  hi6421v530_regulator_driver_init+0x0/0x30 @ 1
[    0.135962] initcall hi6421v530_regulator_driver_init+0x0/0x30 returned 0 after 25 usecs
[    0.135974] calling  max77620_regulator_driver_init+0x0/0x30 @ 1
[    0.136008] initcall max77620_regulator_driver_init+0x0/0x30 returned 0 after 23 usecs
[    0.136020] calling  qcom_spmi_regulator_driver_init+0x0/0x30 @ 1
[    0.136114] initcall qcom_spmi_regulator_driver_init+0x0/0x30 returned 0 after 82 usecs
[    0.136126] calling  pfuze_driver_init+0x0/0x30 @ 1
[    0.136153] initcall pfuze_driver_init+0x0/0x30 returned 0 after 17 usecs
[    0.136164] calling  pwm_regulator_driver_init+0x0/0x30 @ 1
[    0.136210] initcall pwm_regulator_driver_init+0x0/0x30 returned 0 after 36 usecs
[    0.136221] calling  rk808_regulator_driver_init+0x0/0x30 @ 1
[    0.136256] initcall rk808_regulator_driver_init+0x0/0x30 returned 0 after 25 usecs
[    0.136268] calling  s2mps11_pmic_driver_init+0x0/0x30 @ 1
[    0.136303] initcall s2mps11_pmic_driver_init+0x0/0x30 returned 0 after 25 usecs
[    0.136314] calling  brcm_rescal_reset_driver_init+0x0/0x30 @ 1
[    0.136360] initcall brcm_rescal_reset_driver_init+0x0/0x30 returned 0 after 35 usecs
[    0.136372] calling  n_null_init+0x0/0x38 @ 1
[    0.136382] initcall n_null_init+0x0/0x38 returned 0 after 0 usecs
[    0.136391] calling  pty_init+0x0/0x380 @ 1
[    0.137691] initcall pty_init+0x0/0x380 returned 0 after 1260 usecs
[    0.137707] calling  sysrq_init+0x0/0x90 @ 1
[    0.137730] initcall sysrq_init+0x0/0x90 returned 0 after 14 usecs
[    0.137741] calling  xen_hvc_init+0x0/0x298 @ 1
[    0.137750] initcall xen_hvc_init+0x0/0x298 returned -19 after 0 usecs
[    0.137760] calling  serial8250_init+0x0/0x16c @ 1
[    0.137770] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.138357] initcall serial8250_init+0x0/0x16c returned 0 after 573 usecs
[    0.138372] calling  serial_pci_driver_init+0x0/0x38 @ 1
[    0.138412] initcall serial_pci_driver_init+0x0/0x38 returned 0 after 30 usecs
[    0.138424] calling  exar_pci_driver_init+0x0/0x38 @ 1
[    0.138447] initcall exar_pci_driver_init+0x0/0x38 returned 0 after 13 usecs
[    0.138458] calling  fsl8250_platform_driver_init+0x0/0x30 @ 1
[    0.138496] initcall fsl8250_platform_driver_init+0x0/0x30 returned 0 after 26 usecs
[    0.138508] calling  dw8250_platform_driver_init+0x0/0x30 @ 1
[    0.138581] initcall dw8250_platform_driver_init+0x0/0x30 returned 0 after 62 usecs
[    0.138594] calling  of_platform_serial_driver_init+0x0/0x30 @ 1
[    0.138869] 21c0500.serial: ttyS0 at MMIO 0x21c0500 (irq = 24, base_baud = 12500000) is a 16550A
[    7.542134] printk: console [ttyS0] enabled
[    7.546792] 21c0600.serial: ttyS1 at MMIO 0x21c0600 (irq = 24, base_baud = 12500000) is a 16550A
[    7.555855] initcall of_platform_serial_driver_init+0x0/0x30 returned 0 after 7243409 usecs
[    7.564262] calling  lpuart_serial_init+0x0/0x6c @ 1
[    7.569323] platform 2270000.serial: probe deferral - supplier 22c0000.dma-controller not ready
[    7.578109] initcall lpuart_serial_init+0x0/0x6c returned 0 after 8648 usecs
[    7.585195] calling  init+0x0/0x124 @ 1
[    7.589089] initcall init+0x0/0x124 returned 0 after 37 usecs
[    7.594864] calling  tpm_tis_i2c_driver_init+0x0/0x30 @ 1
[    7.600318] initcall tpm_tis_i2c_driver_init+0x0/0x30 returned 0 after 28 usecs
[    7.607664] calling  arm_smmu_driver_init+0x0/0x30 @ 1
[    7.613346] arm-smmu 5000000.iommu: probing hardware configuration...
[    7.619819] arm-smmu 5000000.iommu: SMMUv2 with:
[    7.624460] arm-smmu 5000000.iommu: 	stage 1 translation
[    7.629795] arm-smmu 5000000.iommu: 	stage 2 translation
[    7.635130] arm-smmu 5000000.iommu: 	nested translation
[    7.640379] arm-smmu 5000000.iommu: 	stream matching with 128 register groups
[    7.647549] arm-smmu 5000000.iommu: 	64 context banks (0 stage-2 only)
[    7.654108] arm-smmu 5000000.iommu: 	Supported page sizes: 0x61311000
[    7.660577] arm-smmu 5000000.iommu: 	Stage-1: 48-bit VA -> 48-bit IPA
[    7.667047] arm-smmu 5000000.iommu: 	Stage-2: 48-bit IPA -> 48-bit PA
[    7.673805] initcall arm_smmu_driver_init+0x0/0x30 returned 0 after 59547 usecs
[    7.681153] calling  arm_smmu_driver_init+0x0/0x34 @ 1
[    7.686361] initcall arm_smmu_driver_init+0x0/0x34 returned 0 after 40 usecs
[    7.693445] calling  drm_kms_helper_init+0x0/0x2c @ 1
[    7.698522] initcall drm_kms_helper_init+0x0/0x2c returned 0 after 0 usecs
[    7.705428] calling  drm_core_init+0x0/0xc8 @ 1
[    7.710000] initcall drm_core_init+0x0/0xc8 returned 0 after 18 usecs
[    7.716474] calling  topology_sysfs_init+0x0/0x48 @ 1
[    7.721584] initcall topology_sysfs_init+0x0/0x48 returned 0 after 33 usecs
[    7.728581] calling  cacheinfo_sysfs_init+0x0/0x48 @ 1
[    7.733885] initcall cacheinfo_sysfs_init+0x0/0x48 returned 0 after 137 usecs
[    7.741061] calling  devcoredump_init+0x0/0x38 @ 1
[    7.745885] initcall devcoredump_init+0x0/0x38 returned 0 after 10 usecs
[    7.752624] calling  loop_init+0x0/0x160 @ 1
[    7.760873] loop: module loaded
[    7.764056] initcall loop_init+0x0/0x160 returned 0 after 6972 usecs
[    7.770451] calling  init+0x0/0xac @ 1
[    7.774255] initcall init+0x0/0xac returned 0 after 35 usecs
[    7.779944] calling  xlblk_init+0x0/0x158 @ 1
[    7.784323] initcall xlblk_init+0x0/0x158 returned -19 after 0 usecs
[    7.790708] calling  at24_init+0x0/0x6c @ 1
[    7.795017] at24 0-0050: supply vcc not found, using dummy regulator
[    7.801466] at24 0-0050: Linked as a consumer to regulator.0
[    7.807945] at24 0-0050: 4096 byte 24c32 EEPROM, writable, 32 bytes/write
[    7.814852] at24 1-0057: supply vcc not found, using dummy regulator
[    7.821300] at24 1-0057: Linked as a consumer to regulator.0
[    7.827752] at24 1-0057: 4096 byte 24c32 EEPROM, writable, 32 bytes/write
[    7.834649] at24 2-0050: supply vcc not found, using dummy regulator
[    7.841114] at24 2-0050: Linked as a consumer to regulator.0
[    7.847564] at24 2-0050: 4096 byte 24c32 EEPROM, writable, 32 bytes/write
[    7.854427] initcall at24_init+0x0/0x6c returned 0 after 58118 usecs
[    7.860825] calling  at25_driver_init+0x0/0x30 @ 1
[    7.865655] initcall at25_driver_init+0x0/0x30 returned 0 after 11 usecs
[    7.872392] calling  bd9571mwv_driver_init+0x0/0x30 @ 1
[    7.877668] initcall bd9571mwv_driver_init+0x0/0x30 returned 0 after 21 usecs
[    7.884840] calling  cros_ec_dev_init+0x0/0x94 @ 1
[    7.889733] initcall cros_ec_dev_init+0x0/0x94 returned 0 after 76 usecs
[    7.896470] calling  axp20x_i2c_driver_init+0x0/0x30 @ 1
[    7.901831] initcall axp20x_i2c_driver_init+0x0/0x30 returned 0 after 22 usecs
[    7.909090] calling  max77620_driver_init+0x0/0x30 @ 1
[    7.914270] initcall max77620_driver_init+0x0/0x30 returned 0 after 15 usecs
[    7.921354] calling  rk808_i2c_driver_init+0x0/0x30 @ 1
[    7.926625] initcall rk808_i2c_driver_init+0x0/0x30 returned 0 after 20 usecs
[    7.933796] calling  vexpress_sysreg_driver_init+0x0/0x30 @ 1
[    7.939630] initcall vexpress_sysreg_driver_init+0x0/0x30 returned 0 after 54 usecs
[    7.947326] calling  hi6421_pmic_driver_init+0x0/0x30 @ 1
[    7.952799] initcall hi6421_pmic_driver_init+0x0/0x30 returned 0 after 44 usecs
[    7.960146] calling  simple_mfd_i2c_driver_init+0x0/0x30 @ 1
[    7.966642] platform leds: Linked as a sync state only consumer to 2000000.i2c:sl28cpld@4a:gpio@10
[    7.975682] platform 2000000.i2c:sl28cpld@4a:gpio@10: Linked as a consumer to 2310000.gpio
[    7.989839] platform leds: Linked as a sync state only consumer to 2000000.i2c:sl28cpld@4a:gpio@15
[    7.998885] platform 2000000.i2c:sl28cpld@4a:gpio@15: Linked as a consumer to 2310000.gpio
[    8.013134] platform buttons1: Linked as a sync state only consumer to 2000000.i2c:sl28cpld@4a:gpio@1b
[    8.022758] platform buttons0: Linked as a sync state only consumer to 2000000.i2c:sl28cpld@4a:interrupt-controller@1c
[    8.033540] platform 2000000.i2c:sl28cpld@4a:interrupt-controller@1c: Linked as a consumer to 2310000.gpio
[    8.044098] initcall simple_mfd_i2c_driver_init+0x0/0x30 returned 0 after 76429 usecs
[    8.051995] calling  sas_transport_init+0x0/0xc8 @ 1
[    8.057034] initcall sas_transport_init+0x0/0xc8 returned 0 after 42 usecs
[    8.063961] calling  sas_class_init+0x0/0x94 @ 1
[    8.068620] initcall sas_class_init+0x0/0x94 returned 0 after 15 usecs
[    8.075181] calling  ufshcd_core_init+0x0/0x28 @ 1
[    8.080002] initcall ufshcd_core_init+0x0/0x28 returned 0 after 6 usecs
[    8.086650] calling  init_sd+0x0/0x1a4 @ 1
[    8.090805] initcall init_sd+0x0/0x1a4 returned 0 after 37 usecs
[    8.096843] calling  hisi_sas_init+0x0/0x98 @ 1
[    8.101400] initcall hisi_sas_init+0x0/0x98 returned 0 after 3 usecs
[    8.107785] calling  hisi_sas_v1_driver_init+0x0/0x30 @ 1
[    8.113304] initcall hisi_sas_v1_driver_init+0x0/0x30 returned 0 after 91 usecs
[    8.120651] calling  hisi_sas_v2_driver_init+0x0/0x30 @ 1
[    8.126133] initcall hisi_sas_v2_driver_init+0x0/0x30 returned 0 after 52 usecs
[    8.133480] calling  sas_v3_pci_driver_init+0x0/0x38 @ 1
[    8.138856] initcall sas_v3_pci_driver_init+0x0/0x38 returned 0 after 38 usecs
[    8.146117] calling  ahci_pci_driver_init+0x0/0x38 @ 1
[    8.151298] initcall ahci_pci_driver_init+0x0/0x38 returned 0 after 18 usecs
[    8.158383] calling  ahci_driver_init+0x0/0x30 @ 1
[    8.163275] initcall ahci_driver_init+0x0/0x30 returned 0 after 75 usecs
[    8.170010] calling  sil24_pci_driver_init+0x0/0x38 @ 1
[    8.175279] initcall sil24_pci_driver_init+0x0/0x38 returned 0 after 18 usecs
[    8.182452] calling  ceva_ahci_driver_init+0x0/0x30 @ 1
[    8.187751] initcall ceva_ahci_driver_init+0x0/0x30 returned 0 after 44 usecs
[    8.194923] calling  xgene_ahci_driver_init+0x0/0x30 @ 1
[    8.200316] initcall xgene_ahci_driver_init+0x0/0x30 returned 0 after 53 usecs
[    8.207576] calling  ahci_qoriq_driver_init+0x0/0x30 @ 1
[    8.213015] ahci-qoriq 3200000.sata: Adding to iommu group 0
[    8.218831] ahci-qoriq 3200000.sata: supply ahci not found, using dummy regulator
[    8.226431] ahci-qoriq 3200000.sata: Linked as a consumer to regulator.0
[    8.233171] ahci-qoriq 3200000.sata: supply phy not found, using dummy regulator
[    8.240633] ahci-qoriq 3200000.sata: supply target not found, using dummy regulator
[    8.248414] ahci-qoriq 3200000.sata: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x1 impl platform mode
[    8.257423] ahci-qoriq 3200000.sata: flags: 64bit ncq sntf pm clo only pmp fbs pio slum part ccc sds apst 
[    8.267588] scsi host0: ahci-qoriq
[    8.271166] ata1: SATA max UDMA/133 mmio [mem 0x03200000-0x0320ffff] port 0x100 irq 31
[    8.279248] initcall ahci_qoriq_driver_init+0x0/0x30 returned 0 after 64780 usecs
[    8.286781] calling  pata_platform_driver_init+0x0/0x30 @ 1
[    8.292434] initcall pata_platform_driver_init+0x0/0x30 returned 0 after 49 usecs
[    8.299956] calling  pata_of_platform_driver_init+0x0/0x30 @ 1
[    8.305871] initcall pata_of_platform_driver_init+0x0/0x30 returned 0 after 52 usecs
[    8.313659] calling  init_mtd+0x0/0x138 @ 1
[    8.317945] initcall init_mtd+0x0/0x138 returned 0 after 79 usecs
[    8.324075] calling  ofpart_parser_init+0x0/0x4c @ 1
[    8.329066] initcall ofpart_parser_init+0x0/0x4c returned 0 after 0 usecs
[    8.335888] calling  init_mtdblock+0x0/0x2c @ 1
[    8.340444] initcall init_mtdblock+0x0/0x2c returned 0 after 3 usecs
[    8.346829] calling  denali_dt_driver_init+0x0/0x30 @ 1
[    8.352143] initcall denali_dt_driver_init+0x0/0x30 returned 0 after 63 usecs
[    8.359317] calling  spi_nor_driver_init+0x0/0x30 @ 1
[    8.364408] initcall spi_nor_driver_init+0x0/0x30 returned 0 after 14 usecs
[    8.371409] calling  fsl_dspi_driver_init+0x0/0x30 @ 1
[    8.376611] platform 2120000.spi: probe deferral - supplier 22c0000.dma-controller not ready
[    8.385166] initcall fsl_dspi_driver_init+0x0/0x30 returned 0 after 8391 usecs
[    8.392426] calling  nxp_fspi_driver_init+0x0/0x30 @ 1
[    8.397950] spi-nor spi0.0: w25q32dw (4096 Kbytes)
[    8.403027] 8 fixed-partitions partitions found on MTD device 20c0000.spi
[    8.409854] Creating 8 MTD partitions on "20c0000.spi":
[    8.415115] 0x000000000000-0x000000010000 : "rcw"
[    8.422337] 0x000000010000-0x000000100000 : "failsafe bootloader"
[    8.430296] 0x000000100000-0x000000140000 : "failsafe DP firmware"
[    8.438280] 0x000000140000-0x0000001e0000 : "failsafe trusted firmware"
[    8.446296] 0x0000001e0000-0x000000200000 : "reserved"
[    8.454284] 0x000000200000-0x000000210000 : "configuration store"
[    8.462302] 0x000000210000-0x0000003e0000 : "bootloader"
[    8.470291] 0x0000003e0000-0x000000400000 : "bootloader environment"
[    8.478423] initcall nxp_fspi_driver_init+0x0/0x30 returned 0 after 78937 usecs
[    8.485790] calling  rockchip_spi_driver_init+0x0/0x30 @ 1
[    8.491433] initcall rockchip_spi_driver_init+0x0/0x30 returned 0 after 124 usecs
[    8.498956] calling  net_olddevs_init+0x0/0xa0 @ 1
[    8.503778] initcall net_olddevs_init+0x0/0xa0 returned 0 after 6 usecs
[    8.510429] calling  blackhole_netdev_init+0x0/0x9c @ 1
[    8.515692] initcall blackhole_netdev_init+0x0/0x9c returned 0 after 11 usecs
[    8.522865] calling  phy_module_init+0x0/0x34 @ 1
[    8.527645] initcall phy_module_init+0x0/0x34 returned 0 after 51 usecs
[    8.534295] calling  phy_module_init+0x0/0x34 @ 1
[    8.539036] initcall phy_module_init+0x0/0x34 returned 0 after 13 usecs
[    8.545683] calling  fixed_mdio_bus_init+0x0/0x124 @ 1
[    8.551098] libphy: Fixed MDIO Bus: probed
[    8.555213] initcall fixed_mdio_bus_init+0x0/0x124 returned 0 after 4264 usecs
[    8.562472] calling  phy_module_init+0x0/0x34 @ 1
[    8.567335] initcall phy_module_init+0x0/0x34 returned 0 after 132 usecs
[    8.574076] calling  phy_module_init+0x0/0x34 @ 1
[    8.578944] initcall phy_module_init+0x0/0x34 returned 0 after 136 usecs
[    8.585679] calling  phy_module_init+0x0/0x34 @ 1
[    8.590415] initcall phy_module_init+0x0/0x34 returned 0 after 10 usecs
[    8.597062] calling  thunder_mdiobus_driver_init+0x0/0x38 @ 1
[    8.600144] ata1: SATA link down (SStatus 0 SControl 300)
[    8.602874] initcall thunder_mdiobus_driver_init+0x0/0x38 returned 0 after 36 usecs
[    8.615949] calling  mdio_mux_mmioreg_driver_init+0x0/0x30 @ 1
[    8.621877] initcall mdio_mux_mmioreg_driver_init+0x0/0x30 returned 0 after 64 usecs
[    8.629661] calling  tun_init+0x0/0xd4 @ 1
[    8.633777] tun: Universal TUN/TAP device driver, 1.6
[    8.638936] initcall tun_init+0x0/0xd4 returned 0 after 5038 usecs
[    8.645152] calling  virtio_net_driver_init+0x0/0xc0 @ 1
[    8.650507] initcall virtio_net_driver_init+0x0/0xc0 returned 0 after 17 usecs
[    8.657767] calling  can_dev_init+0x0/0x44 @ 1
[    8.662233] CAN device driver interface
[    8.666082] initcall can_dev_init+0x0/0x44 returned 0 after 3759 usecs
[    8.672640] calling  flexcan_driver_init+0x0/0x30 @ 1
[    8.678221] initcall flexcan_driver_init+0x0/0x30 returned 0 after 493 usecs
[    8.685314] calling  felix_vsc9959_pci_driver_init+0x0/0x38 @ 1
[    8.691298] initcall felix_vsc9959_pci_driver_init+0x0/0x38 returned 0 after 30 usecs
[    8.699173] calling  xgbe_mod_init+0x0/0x68 @ 1
[    8.703804] initcall xgbe_mod_init+0x0/0x68 returned 0 after 74 usecs
[    8.710279] calling  macb_driver_init+0x0/0x30 @ 1
[    8.715236] initcall macb_driver_init+0x0/0x30 returned 0 after 138 usecs
[    8.722060] calling  xcv_init_module+0x0/0x5c @ 1
[    8.726788] thunder_xcv, ver 1.0
[    8.730046] initcall xcv_init_module+0x0/0x5c returned 0 after 3182 usecs
[    8.736869] calling  bgx_init_module+0x0/0x5c @ 1
[    8.741596] thunder_bgx, ver 1.0
[    8.744858] initcall bgx_init_module+0x0/0x5c returned 0 after 3184 usecs
[    8.751680] calling  nic_init_module+0x0/0x5c @ 1
[    8.756407] nicpf, ver 1.0
[    8.759144] initcall nic_init_module+0x0/0x5c returned 0 after 2673 usecs
[    8.765969] calling  enetc_pf_driver_init+0x0/0x38 @ 1
[    8.771149] initcall enetc_pf_driver_init+0x0/0x38 returned 0 after 15 usecs
[    8.778234] calling  enetc_pci_mdio_driver_init+0x0/0x38 @ 1
[    8.783937] initcall enetc_pci_mdio_driver_init+0x0/0x38 returned 0 after 16 usecs
[    8.791546] calling  hix5hd2_dev_driver_init+0x0/0x30 @ 1
[    8.797043] initcall hix5hd2_dev_driver_init+0x0/0x30 returned 0 after 70 usecs
[    8.804391] calling  hns_mdio_driver_init+0x0/0x30 @ 1
[    8.809611] initcall hns_mdio_driver_init+0x0/0x30 returned 0 after 54 usecs
[    8.816697] calling  g_dsaf_driver_init+0x0/0x30 @ 1
[    8.821740] initcall g_dsaf_driver_init+0x0/0x30 returned 0 after 50 usecs
[    8.828650] calling  hns_nic_dev_driver_init+0x0/0x30 @ 1
[    8.834145] initcall hns_nic_dev_driver_init+0x0/0x30 returned 0 after 62 usecs
[    8.841495] calling  hclge_init+0x0/0x8c @ 1
[    8.845790] hclge is initializing
[    8.849126] initcall hclge_init+0x0/0x8c returned 0 after 3258 usecs
[    8.855511] calling  hns3_init_module+0x0/0xe4 @ 1
[    8.860326] hns3: Hisilicon Ethernet Network Driver for Hip08 Family - version
[    8.867580] hns3: Copyright (c) 2017 Huawei Corporation.
[    8.872943] initcall hns3_init_module+0x0/0xe4 returned 0 after 12321 usecs
[    8.879946] calling  igb_init_module+0x0/0x68 @ 1
[    8.884674] igb: Intel(R) Gigabit Ethernet Network Driver
[    8.890094] igb: Copyright (c) 2007-2014 Intel Corporation.
[    8.895706] initcall igb_init_module+0x0/0x68 returned 0 after 10773 usecs
[    8.902616] calling  orion_mdio_driver_init+0x0/0x30 @ 1
[    8.908007] initcall orion_mdio_driver_init+0x0/0x30 returned 0 after 52 usecs
[    8.915268] calling  sky2_init_module+0x0/0x44 @ 1
[    8.920082] sky2: driver version 1.30
[    8.923778] initcall sky2_init_module+0x0/0x44 returned 0 after 3609 usecs
[    8.930688] calling  mscc_ocelot_driver_init+0x0/0x30 @ 1
[    8.936161] initcall mscc_ocelot_driver_init+0x0/0x30 returned 0 after 46 usecs
[    8.943511] calling  rtl8169_pci_driver_init+0x0/0x38 @ 1
[    8.948953] initcall rtl8169_pci_driver_init+0x0/0x38 returned 0 after 14 usecs
[    8.956300] calling  smc_driver_init+0x0/0x30 @ 1
[    8.961080] initcall smc_driver_init+0x0/0x30 returned 0 after 51 usecs
[    8.967729] calling  smsc911x_init_module+0x0/0x30 @ 1
[    8.972937] initcall smsc911x_init_module+0x0/0x30 returned 0 after 43 usecs
[    8.980023] calling  stmmac_init+0x0/0x60 @ 1
[    8.984408] initcall stmmac_init+0x0/0x60 returned 0 after 6 usecs
[    8.990623] calling  netif_init+0x0/0x7c @ 1
[    8.994915] initcall netif_init+0x0/0x7c returned -19 after 0 usecs
[    9.001213] calling  net_failover_init+0x0/0x18 @ 1
[    9.006118] initcall net_failover_init+0x0/0x18 returned 0 after 0 usecs
[    9.012853] calling  vfio_init+0x0/0x18c @ 1
[    9.017232] VFIO - User Level meta-driver version: 0.3
[    9.022396] initcall vfio_init+0x0/0x18c returned 0 after 5128 usecs
[    9.028782] calling  vfio_virqfd_init+0x0/0x54 @ 1
[    9.033691] initcall vfio_virqfd_init+0x0/0x54 returned 0 after 92 usecs
[    9.040439] calling  vfio_iommu_type1_init+0x0/0x2c @ 1
[    9.045696] initcall vfio_iommu_type1_init+0x0/0x2c returned 0 after 1 usecs
[    9.052783] calling  vfio_pci_init+0x0/0x198 @ 1
[    9.057463] initcall vfio_pci_init+0x0/0x198 returned 0 after 38 usecs
[    9.064026] calling  usb_conn_driver_init+0x0/0x30 @ 1
[    9.069257] initcall usb_conn_driver_init+0x0/0x30 returned 0 after 64 usecs
[    9.076343] calling  dwc3_driver_init+0x0/0x30 @ 1
[    9.081236] dwc3 3100000.usb: Adding to iommu group 1
[    9.086752] dwc3 3110000.usb: Adding to iommu group 2
[    9.092203] initcall dwc3_driver_init+0x0/0x30 returned 0 after 10785 usecs
[    9.099211] calling  dwc3_pci_driver_init+0x0/0x38 @ 1
[    9.104411] initcall dwc3_pci_driver_init+0x0/0x38 returned 0 after 33 usecs
[    9.111496] calling  dwc3_haps_driver_init+0x0/0x38 @ 1
[    9.116765] initcall dwc3_haps_driver_init+0x0/0x38 returned 0 after 15 usecs
[    9.123940] calling  dwc3_of_simple_driver_init+0x0/0x30 @ 1
[    9.129729] initcall dwc3_of_simple_driver_init+0x0/0x30 returned 0 after 98 usecs
[    9.137338] calling  dwc2_platform_driver_init+0x0/0x30 @ 1
[    9.143083] initcall dwc2_platform_driver_init+0x0/0x30 returned 0 after 140 usecs
[    9.150697] calling  isp1760_init+0x0/0x7c @ 1
[    9.155383] initcall isp1760_init+0x0/0x7c returned 0 after 213 usecs
[    9.161860] calling  ehci_hcd_init+0x0/0xa8 @ 1
[    9.166415] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    9.172971] initcall ehci_hcd_init+0x0/0xa8 returned 0 after 6402 usecs
[    9.179618] calling  ehci_pci_init+0x0/0x98 @ 1
[    9.184171] ehci-pci: EHCI PCI platform driver
[    9.188648] initcall ehci_pci_init+0x0/0x98 returned 0 after 4372 usecs
[    9.195298] calling  ehci_platform_init+0x0/0x70 @ 1
[    9.200289] ehci-platform: EHCI generic platform driver
[    9.205603] initcall ehci_platform_init+0x0/0x70 returned 0 after 5189 usecs
[    9.212692] calling  ohci_hcd_mod_init+0x0/0xb0 @ 1
[    9.217596] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    9.223810] initcall ohci_hcd_mod_init+0x0/0xb0 returned 0 after 6068 usecs
[    9.230809] calling  ohci_pci_init+0x0/0x98 @ 1
[    9.235362] ohci-pci: OHCI PCI platform driver
[    9.239840] initcall ohci_pci_init+0x0/0x98 returned 0 after 4373 usecs
[    9.246497] calling  ohci_platform_init+0x0/0x70 @ 1
[    9.251488] ohci-platform: OHCI generic platform driver
[    9.256799] initcall ohci_platform_init+0x0/0x70 returned 0 after 5185 usecs
[    9.263884] calling  xhci_hcd_init+0x0/0x44 @ 1
[    9.268441] initcall xhci_hcd_init+0x0/0x44 returned 0 after 4 usecs
[    9.274829] calling  xhci_pci_init+0x0/0x78 @ 1
[    9.279400] initcall xhci_pci_init+0x0/0x78 returned 0 after 18 usecs
[    9.285871] calling  xhci_plat_init+0x0/0x44 @ 1
[    9.290697] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[    9.296220] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
[    9.304091] xhci-hcd xhci-hcd.0.auto: hcc params 0x0220f66d hci version 0x100 quirks 0x0000000002010010
[    9.313556] xhci-hcd xhci-hcd.0.auto: irq 29, io mem 0x03100000
[    9.319917] hub 1-0:1.0: USB hub found
[    9.323701] hub 1-0:1.0: 1 port detected
[    9.327802] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[    9.333321] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
[    9.341022] xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0 SuperSpeed
[    9.347623] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    9.356004] hub 2-0:1.0: USB hub found
[    9.359785] hub 2-0:1.0: 1 port detected
[    9.363943] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
[    9.369466] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 3
[    9.377335] xhci-hcd xhci-hcd.1.auto: hcc params 0x0220f66d hci version 0x100 quirks 0x0000000002010010
[    9.386800] xhci-hcd xhci-hcd.1.auto: irq 30, io mem 0x03110000
[    9.393117] hub 3-0:1.0: USB hub found
[    9.396900] hub 3-0:1.0: 1 port detected
[    9.400990] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
[    9.406509] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 4
[    9.414206] xhci-hcd xhci-hcd.1.auto: Host supports USB 3.0 SuperSpeed
[    9.420799] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    9.429174] hub 4-0:1.0: USB hub found
[    9.432965] hub 4-0:1.0: 1 port detected
[    9.437083] initcall xhci_plat_init+0x0/0x44 returned 0 after 143138 usecs
[    9.444001] calling  usb_storage_driver_init+0x0/0x54 @ 1
[    9.449487] usbcore: registered new interface driver usb-storage
[    9.455522] initcall usb_storage_driver_init+0x0/0x54 returned 0 after 5953 usecs
[    9.463044] calling  usb3503_init+0x0/0x74 @ 1
[    9.467662] initcall usb3503_init+0x0/0x74 returned 0 after 143 usecs
[    9.474137] calling  musb_driver_init+0x0/0x34 @ 1
[    9.478982] initcall musb_driver_init+0x0/0x34 returned 0 after 31 usecs
[    9.485716] calling  ci_hdrc_platform_register+0x0/0x38 @ 1
[    9.491345] initcall ci_hdrc_platform_register+0x0/0x38 returned 0 after 30 usecs
[    9.498864] calling  ci_hdrc_usb2_driver_init+0x0/0x30 @ 1
[    9.504444] initcall ci_hdrc_usb2_driver_init+0x0/0x30 returned 0 after 60 usecs
[    9.511878] calling  ci_hdrc_msm_driver_init+0x0/0x30 @ 1
[    9.517349] initcall ci_hdrc_msm_driver_init+0x0/0x30 returned 0 after 45 usecs
[    9.524696] calling  ci_hdrc_pci_driver_init+0x0/0x38 @ 1
[    9.530148] initcall ci_hdrc_pci_driver_init+0x0/0x38 returned 0 after 27 usecs
[    9.537494] calling  ci_hdrc_imx_driver_init+0x0/0x30 @ 1
[    9.543026] initcall ci_hdrc_imx_driver_init+0x0/0x30 returned 0 after 101 usecs
[    9.550458] calling  usbmisc_imx_driver_init+0x0/0x30 @ 1
[    9.555996] initcall usbmisc_imx_driver_init+0x0/0x30 returned 0 after 111 usecs
[    9.563428] calling  tegra_usb_driver_init+0x0/0x30 @ 1
[    9.568756] initcall tegra_usb_driver_init+0x0/0x30 returned 0 after 77 usecs
[    9.575925] calling  gadget_cfs_init+0x0/0x40 @ 1
[    9.580698] initcall gadget_cfs_init+0x0/0x40 returned 0 after 43 usecs
[    9.587350] calling  udc_plat_driver_init+0x0/0x30 @ 1
[    9.592577] initcall udc_plat_driver_init+0x0/0x30 returned 0 after 62 usecs
[    9.599660] calling  bdc_driver_init+0x0/0x30 @ 1
[    9.604439] initcall bdc_driver_init+0x0/0x30 returned 0 after 52 usecs
[    9.611097] calling  bdc_pci_driver_init+0x0/0x38 @ 1
[    9.616197] initcall bdc_pci_driver_init+0x0/0x38 returned 0 after 23 usecs
[    9.623192] calling  acmmod_init+0x0/0x2c @ 1
[    9.627570] initcall acmmod_init+0x0/0x2c returned 0 after 0 usecs
[    9.633777] calling  userial_init+0x0/0x170 @ 1
[    9.638338] initcall userial_init+0x0/0x170 returned 0 after 3 usecs
[    9.644722] calling  gsermod_init+0x0/0x2c @ 1
[    9.649189] initcall gsermod_init+0x0/0x2c returned 0 after 0 usecs
[    9.655485] calling  obexmod_init+0x0/0x2c @ 1
[    9.659949] initcall obexmod_init+0x0/0x2c returned 0 after 0 usecs
[    9.666253] calling  ecmmod_init+0x0/0x2c @ 1
[    9.670631] initcall ecmmod_init+0x0/0x2c returned 0 after 0 usecs
[    9.676839] calling  gethmod_init+0x0/0x2c @ 1
[    9.681302] initcall gethmod_init+0x0/0x2c returned 0 after 0 usecs
[    9.687597] calling  rndismod_init+0x0/0x2c @ 1
[    9.692149] initcall rndismod_init+0x0/0x2c returned 0 after 0 usecs
[    9.698531] calling  mass_storagemod_init+0x0/0x30 @ 1
[    9.703694] initcall mass_storagemod_init+0x0/0x30 returned 0 after 0 usecs
[    9.710691] calling  eth_driver_init+0x0/0x2c @ 1
[    9.715418] udc-core: couldn't find an available UDC - added [g_ether] to list of pending drivers
[    9.724331] initcall eth_driver_init+0x0/0x2c returned 0 after 8705 usecs
[    9.731150] calling  msg_driver_init+0x0/0x2c @ 1
[    9.735875] udc-core: couldn't find an available UDC - added [g_mass_storage] to list of pending drivers
[    9.737943] usb 3-1: new high-speed USB device number 2 using xhci-hcd
[    9.745398] initcall msg_driver_init+0x0/0x2c returned 0 after 9299 usecs
[    9.758774] calling  init+0x0/0xd4 @ 1
[    9.762566] udc-core: couldn't find an available UDC - added [g_serial] to list of pending drivers
[    9.771575] initcall init+0x0/0xd4 returned 0 after 8798 usecs
[    9.777439] calling  ambakmi_driver_init+0x0/0x2c @ 1
[    9.782531] initcall ambakmi_driver_init+0x0/0x2c returned 0 after 16 usecs
[    9.789525] calling  input_leds_init+0x0/0x2c @ 1
[    9.794252] initcall input_leds_init+0x0/0x2c returned 0 after 0 usecs
[    9.800810] calling  evdev_init+0x0/0x2c @ 1
[    9.805098] initcall evdev_init+0x0/0x2c returned 0 after 0 usecs
[    9.811219] calling  atkbd_init+0x0/0x48 @ 1
[    9.815547] initcall atkbd_init+0x0/0x48 returned 0 after 32 usecs
[    9.821762] calling  cros_ec_keyb_driver_init+0x0/0x30 @ 1
[    9.827366] initcall cros_ec_keyb_driver_init+0x0/0x30 returned 0 after 79 usecs
[    9.834803] calling  gpio_keys_polled_driver_init+0x0/0x30 @ 1
[    9.840824] input: buttons1 as /devices/platform/buttons1/input/input0
[    9.848159] initcall gpio_keys_polled_driver_init+0x0/0x30 returned 0 after 7319 usecs
[    9.856136] calling  psmouse_init+0x0/0xa4 @ 1
[    9.860659] initcall psmouse_init+0x0/0xa4 returned 0 after 49 usecs
[    9.867057] calling  xenkbd_init+0x0/0x58 @ 1
[    9.871436] initcall xenkbd_init+0x0/0x58 returned -19 after 0 usecs
[    9.877819] calling  cros_ec_rtc_driver_init+0x0/0x30 @ 1
[    9.883287] initcall cros_ec_rtc_driver_init+0x0/0x30 returned 0 after 43 usecs
[    9.890641] calling  ds323x_init+0x0/0x88 @ 1
[    9.895070] initcall ds323x_init+0x0/0x88 returned 0 after 41 usecs
[    9.901371] calling  efi_rtc_driver_init+0x0/0x38 @ 1
[    9.906483] initcall efi_rtc_driver_init+0x0/0x38 returned -19 after 36 usecs
[    9.913654] calling  ftm_alarm_init+0x0/0x30 @ 1
[    9.918744] ftm-alarm 2800000.timer: registered as rtc1
[    9.924062] initcall ftm_alarm_init+0x0/0x30 returned 0 after 5612 usecs
[    9.930801] calling  max77686_rtc_driver_init+0x0/0x30 @ 1
[    9.936372] initcall max77686_rtc_driver_init+0x0/0x30 returned 0 after 50 usecs
[    9.943815] calling  pl031_driver_init+0x0/0x2c @ 1
[    9.948732] initcall pl031_driver_init+0x0/0x2c returned 0 after 14 usecs
[    9.955557] calling  rv8803_driver_init+0x0/0x30 @ 1
[    9.961178] rtc-rv8803 0-0032: Voltage low, temperature compensation stopped.
[    9.968359] rtc-rv8803 0-0032: Voltage low, data loss detected.
[    9.974389] hub 3-1:1.0: USB hub found
[    9.975394] rtc-rv8803 0-0032: Voltage low, data is invalid.
[    9.983867] hub 3-1:1.0: 7 ports detected
[    9.985983] rtc-rv8803 0-0032: registered as rtc0
[    9.993207] rtc-rv8803 0-0032: Voltage low, data is invalid.
[    9.998906] rtc-rv8803 0-0032: hctosys: unable to read the hardware clock
[   10.005869] initcall rv8803_driver_init+0x0/0x30 returned 0 after 44261 usecs
[   10.013049] calling  s5m_rtc_driver_init+0x0/0x30 @ 1
[   10.018186] initcall s5m_rtc_driver_init+0x0/0x30 returned 0 after 54 usecs
[   10.025183] calling  i2c_dev_init+0x0/0xf0 @ 1
[   10.029655] i2c /dev entries driver
[   10.033371] initcall i2c_dev_init+0x0/0xf0 returned 0 after 3629 usecs
[   10.039947] calling  rk3x_i2c_driver_init+0x0/0x30 @ 1
[   10.045261] initcall rk3x_i2c_driver_init+0x0/0x30 returned 0 after 133 usecs
[   10.052453] calling  ec_i2c_tunnel_driver_init+0x0/0x30 @ 1
[   10.058134] initcall ec_i2c_tunnel_driver_init+0x0/0x30 returned 0 after 69 usecs
[   10.065656] calling  pca954x_driver_init+0x0/0x30 @ 1
[   10.070764] initcall pca954x_driver_init+0x0/0x30 returned 0 after 31 usecs
[   10.077764] calling  vexpress_reset_driver_init+0x0/0x30 @ 1
[   10.083519] initcall vexpress_reset_driver_init+0x0/0x30 returned 0 after 63 usecs
[   10.091128] calling  xgene_reboot_init+0x0/0x30 @ 1
[   10.096079] initcall xgene_reboot_init+0x0/0x30 returned 0 after 46 usecs
[   10.102900] calling  syscon_reboot_driver_init+0x0/0x30 @ 1
[   10.108656] initcall syscon_reboot_driver_init+0x0/0x30 returned 0 after 153 usecs
[   10.116267] calling  syscon_reboot_mode_driver_init+0x0/0x30 @ 1
[   10.122351] initcall syscon_reboot_mode_driver_init+0x0/0x30 returned 0 after 48 usecs
[   10.130308] calling  bq27xxx_battery_i2c_driver_init+0x0/0x30 @ 1
[   10.136473] initcall bq27xxx_battery_i2c_driver_init+0x0/0x30 returned 0 after 40 usecs
[   10.144520] calling  scpi_hwmon_platdrv_init+0x0/0x30 @ 1
[   10.150002] initcall scpi_hwmon_platdrv_init+0x0/0x30 returned 0 after 54 usecs
[   10.157348] calling  sl28cpld_hwmon_driver_init+0x0/0x30 @ 1
[   10.163264] initcall sl28cpld_hwmon_driver_init+0x0/0x30 returned 0 after 202 usecs
[   10.170990] calling  qoriq_tmu_init+0x0/0x30 @ 1
[   10.189106] initcall qoriq_tmu_init+0x0/0x30 returned 0 after 13131 usecs
[   10.195939] calling  sp805_wdt_driver_init+0x0/0x2c @ 1
[   10.201364] sp805-wdt c000000.watchdog: registration successful
[   10.207415] sp805-wdt c010000.watchdog: registration successful
[   10.213393] initcall sp805_wdt_driver_init+0x0/0x2c returned 0 after 11916 usecs
[   10.220828] calling  dw_wdt_driver_init+0x0/0x30 @ 1
[   10.225880] initcall dw_wdt_driver_init+0x0/0x30 returned 0 after 61 usecs
[   10.232790] calling  imx2_wdt_driver_init+0x0/0x38 @ 1
[   10.238009] initcall imx2_wdt_driver_init+0x0/0x38 returned -19 after 55 usecs
[   10.245270] calling  sl28cpld_wdt_driver_init+0x0/0x30 @ 1
[   10.252587] sl28cpld-wdt 2000000.i2c:sl28cpld@4a:watchdog@4: initial timeout 6 sec
[   10.260234] initcall sl28cpld_wdt_driver_init+0x0/0x30 returned 0 after 9227 usecs
[   10.267858] calling  dt_cpufreq_platdrv_init+0x0/0x30 @ 1
[   10.273319] initcall dt_cpufreq_platdrv_init+0x0/0x30 returned 0 after 33 usecs
[   10.280689] calling  scpi_cpufreq_platdrv_init+0x0/0x30 @ 1
[   10.286334] initcall scpi_cpufreq_platdrv_init+0x0/0x30 returned 0 after 36 usecs
[   10.293865] calling  qoriq_cpufreq_platform_driver_init+0x0/0x30 @ 1
[   10.300644] qoriq-cpufreq qoriq-cpufreq: Freescale QorIQ CPU frequency scaling driver
[   10.308555] initcall qoriq_cpufreq_platform_driver_init+0x0/0x30 returned 0 after 8110 usecs
[   10.317066] calling  arm_idle_init+0x0/0x190 @ 1
[   10.321730] initcall arm_idle_init+0x0/0x190 returned -95 after 12 usecs
[   10.328466] calling  psci_idle_init+0x0/0xc0 @ 1
[   10.333282] initcall psci_idle_init+0x0/0xc0 returned 0 after 173 usecs
[   10.339934] calling  mmc_pwrseq_simple_driver_init+0x0/0x30 @ 1
[   10.345948] initcall mmc_pwrseq_simple_driver_init+0x0/0x30 returned 0 after 62 usecs
[   10.353818] calling  mmc_pwrseq_emmc_driver_init+0x0/0x30 @ 1
[   10.359644] initcall mmc_pwrseq_emmc_driver_init+0x0/0x30 returned 0 after 50 usecs
[   10.367339] calling  mmc_blk_init+0x0/0x124 @ 1
[   10.371917] initcall mmc_blk_init+0x0/0x124 returned 0 after 24 usecs
[   10.377945] usb 3-1.6: new full-speed USB device number 3 using xhci-hcd
[   10.378391] calling  mmci_driver_init+0x0/0x30 @ 1
[   10.389946] initcall mmci_driver_init+0x0/0x30 returned 0 after 18 usecs
[   10.396703] calling  sdhci_drv_init+0x0/0x3c @ 1
[   10.401369] sdhci: Secure Digital Host Controller Interface driver
[   10.407577] sdhci: Copyright(c) Pierre Ossman
[   10.411951] initcall sdhci_drv_init+0x0/0x3c returned 0 after 10346 usecs
[   10.418773] calling  sdhci_acpi_driver_init+0x0/0x30 @ 1
[   10.424174] initcall sdhci_acpi_driver_init+0x0/0x30 returned 0 after 46 usecs
[   10.431432] calling  sdhci_f_sdh30_driver_init+0x0/0x30 @ 1
[   10.437084] initcall sdhci_f_sdh30_driver_init+0x0/0x30 returned 0 after 50 usecs
[   10.444607] calling  mmc_spi_driver_init+0x0/0x30 @ 1
[   10.449728] initcall mmc_spi_driver_init+0x0/0x30 returned 0 after 15 usecs
[   10.456725] calling  dw_mci_init+0x0/0x30 @ 1
[   10.461103] Synopsys Designware Multimedia Card Interface Driver
[   10.467134] initcall dw_mci_init+0x0/0x30 returned 0 after 5890 usecs
[   10.473605] calling  dw_mci_pltfm_driver_init+0x0/0x30 @ 1
[   10.479182] initcall dw_mci_pltfm_driver_init+0x0/0x30 returned 0 after 65 usecs
[   10.486616] calling  dw_mci_exynos_pltfm_driver_init+0x0/0x30 @ 1
[   10.492825] initcall dw_mci_exynos_pltfm_driver_init+0x0/0x30 returned 0 after 76 usecs
[   10.500877] calling  dw_mci_hi3798cv200_driver_init+0x0/0x30 @ 1
[   10.506967] initcall dw_mci_hi3798cv200_driver_init+0x0/0x30 returned 0 after 49 usecs
[   10.514929] calling  dw_mci_k3_pltfm_driver_init+0x0/0x30 @ 1
[   10.520769] initcall dw_mci_k3_pltfm_driver_init+0x0/0x30 returned 0 after 61 usecs
[   10.528483] calling  sdhci_pltfm_drv_init+0x0/0x30 @ 1
[   10.533663] sdhci-pltfm: SDHCI platform and OF driver helper
[   10.539362] initcall sdhci_pltfm_drv_init+0x0/0x30 returned 0 after 5565 usecs
[   10.546636] calling  sdhci_cdns_driver_init+0x0/0x30 @ 1
[   10.552050] initcall sdhci_cdns_driver_init+0x0/0x30 returned 0 after 57 usecs
[   10.559327] calling  sdhci_arasan_driver_init+0x0/0x30 @ 1
[   10.564965] initcall sdhci_arasan_driver_init+0x0/0x30 returned 0 after 107 usecs
[   10.572491] calling  sdhci_esdhc_driver_init+0x0/0x30 @ 1
[   10.578046] initcall sdhci_esdhc_driver_init+0x0/0x30 returned 0 after 120 usecs
[   10.578050] sdhci-esdhc 2140000.mmc: Adding to iommu group 3
[   10.578284] sdhci-esdhc 2150000.mmc: Adding to iommu group 4
[   10.585484] calling  sdhci_xenon_driver_init+0x0/0x30 @ 1
[   10.602413] initcall sdhci_xenon_driver_init+0x0/0x30 returned 0 after 124 usecs
[   10.609876] calling  gpio_led_driver_init+0x0/0x30 @ 1
[   10.614484] mmc1: SDHCI controller on 2140000.mmc [2140000.mmc] using ADMA
[   10.619830] initcall gpio_led_driver_init+0x0/0x30 returned 0 after 4661 usecs
[   10.621951] mmc0: SDHCI controller on 2150000.mmc [2150000.mmc] using ADMA
[   10.629222] calling  led_pwm_driver_init+0x0/0x30 @ 1
[   10.641399] initcall led_pwm_driver_init+0x0/0x30 returned 0 after 171 usecs
[   10.648543] calling  syscon_led_driver_init+0x0/0x30 @ 1
[   10.653978] initcall syscon_led_driver_init+0x0/0x30 returned 0 after 82 usecs
[   10.661258] calling  ledtrig_disk_init+0x0/0x74 @ 1
[   10.666179] initcall ledtrig_disk_init+0x0/0x74 returned 0 after 14 usecs
[   10.673044] calling  heartbeat_trig_init+0x0/0x5c @ 1
[   10.678189] initcall heartbeat_trig_init+0x0/0x5c returned 0 after 57 usecs
[   10.685267] calling  ledtrig_cpu_init+0x0/0x130 @ 1
[   10.690264] ledtrig-cpu: registered to indicate activity on CPUs
[   10.696366] initcall ledtrig_cpu_init+0x0/0x130 returned 0 after 6042 usecs
[   10.703402] calling  defon_led_trigger_init+0x0/0x2c @ 1
[   10.708811] initcall defon_led_trigger_init+0x0/0x2c returned 0 after 60 usecs
[   10.716123] calling  ledtrig_panic_init+0x0/0x5c @ 1
[   10.721164] initcall ledtrig_panic_init+0x0/0x5c returned 0 after 34 usecs
[   10.728110] calling  scpi_driver_init+0x0/0x30 @ 1
[   10.733010] initcall scpi_driver_init+0x0/0x30 returned 0 after 74 usecs
[   10.739772] calling  scpi_power_domain_driver_init+0x0/0x30 @ 1
[   10.745796] initcall scpi_power_domain_driver_init+0x0/0x30 returned 0 after 60 usecs
[   10.753665] random: fast init done
[   10.757105] calling  esrt_sysfs_init+0x0/0x2b0 @ 1
[   10.761941] initcall esrt_sysfs_init+0x0/0x2b0 returned -38 after 4 usecs
[   10.768812] calling  efivars_pstore_init+0x0/0xd8 @ 1
[   10.773901] initcall efivars_pstore_init+0x0/0xd8 returned 0 after 0 usecs
[   10.780820] calling  efi_capsule_loader_init+0x0/0x68 @ 1
[   10.786247] initcall efi_capsule_loader_init+0x0/0x68 returned -19 after 0 usecs
[   10.793690] calling  smccc_soc_init+0x0/0x27c @ 1
[   10.798424] initcall smccc_soc_init+0x0/0x27c returned 0 after 0 usecs
[   10.804985] calling  meson_crypto_driver_init+0x0/0x30 @ 1
[   10.810589] initcall meson_crypto_driver_init+0x0/0x30 returned 0 after 68 usecs
[   10.818057] calling  hid_init+0x0/0x6c @ 1
[   10.822209] initcall hid_init+0x0/0x6c returned 0 after 22 usecs
[   10.828255] calling  hid_generic_init+0x0/0x38 @ 1
[   10.833114] initcall hid_generic_init+0x0/0x38 returned 0 after 37 usecs
[   10.839860] calling  a4_driver_init+0x0/0x38 @ 1
[   10.844579] initcall a4_driver_init+0x0/0x38 returned 0 after 46 usecs
[   10.851176] calling  apple_driver_init+0x0/0x38 @ 1
[   10.856113] initcall apple_driver_init+0x0/0x38 returned 0 after 24 usecs
[   10.862943] calling  belkin_driver_init+0x0/0x38 @ 1
[   10.867960] initcall belkin_driver_init+0x0/0x38 returned 0 after 16 usecs
[   10.874882] calling  ch_driver_init+0x0/0x38 @ 1
[   10.879555] initcall ch_driver_init+0x0/0x38 returned 0 after 18 usecs
[   10.886145] calling  ch_driver_init+0x0/0x38 @ 1
[   10.890821] initcall ch_driver_init+0x0/0x38 returned 0 after 18 usecs
[   10.897412] calling  cp_driver_init+0x0/0x38 @ 1
[   10.902091] initcall cp_driver_init+0x0/0x38 returned 0 after 23 usecs
[   10.905944] usb 3-1.3: new high-speed USB device number 4 using xhci-hcd
[   10.908678] calling  ez_driver_init+0x0/0x38 @ 1
[   10.920062] initcall ez_driver_init+0x0/0x38 returned 0 after 30 usecs
[   10.926664] calling  ite_driver_init+0x0/0x38 @ 1
[   10.931432] initcall ite_driver_init+0x0/0x38 returned 0 after 18 usecs
[   10.938108] calling  ks_driver_init+0x0/0x38 @ 1
[   10.942781] initcall ks_driver_init+0x0/0x38 returned 0 after 18 usecs
[   10.949352] calling  lg_driver_init+0x0/0x38 @ 1
[   10.954022] initcall lg_driver_init+0x0/0x38 returned 0 after 17 usecs
[   10.960625] calling  lg_g15_driver_init+0x0/0x38 @ 1
[   10.965653] initcall lg_g15_driver_init+0x0/0x38 returned 0 after 19 usecs
[   10.972573] calling  ms_driver_init+0x0/0x38 @ 1
[   10.977243] initcall ms_driver_init+0x0/0x38 returned 0 after 20 usecs
[   10.983816] calling  mr_driver_init+0x0/0x38 @ 1
[   10.988489] initcall mr_driver_init+0x0/0x38 returned 0 after 18 usecs
[   10.989473] mmc1: new ultra high speed SDR104 SDHC card at address 5048
[   10.995050] calling  redragon_driver_init+0x0/0x38 @ 1
[   11.002122] mmcblk1: mmc1:5048 SD16G 14.4 GiB 
[   11.006899] initcall redragon_driver_init+0x0/0x38 returned 0 after 32 usecs
[   11.018481] calling  hid_init+0x0/0x84 @ 1
[   11.022618] mmc0: new HS400 MMC card at address 0001
[   11.023022] mmcblk0: mmc0:0001 S0J58X 29.6 GiB 
[   11.028965] hid-generic 0003:064F:2AF9.0001: device has no listeners, quitting
[   11.032658] mmcblk0boot0: mmc0:0001 S0J58X partition 1 31.5 MiB
[   11.039661] usbcore: registered new interface driver usbhid
[   11.045740] mmcblk0boot1: mmc0:0001 S0J58X partition 2 31.5 MiB
[   11.051044] usbhid: USB HID core driver
[   11.057120] mmcblk0rpmb: mmc0:0001 S0J58X partition 3 4.00 MiB, chardev (241:0)
[   11.060873] initcall hid_init+0x0/0x84 returned 0 after 32476 usecs
[   11.074530] calling  cros_ec_driver_init+0x0/0x30 @ 1
[   11.079688] initcall cros_ec_driver_init+0x0/0x30 returned 0 after 44 usecs
[   11.086761] calling  cros_ec_driver_spi_init+0x0/0x30 @ 1
[   11.086859]  mmcblk0: p1 p2
[   11.092253] initcall cros_ec_driver_spi_init+0x0/0x30 returned 0 after 36 usecs
[   11.102435] calling  cros_ec_chardev_driver_init+0x0/0x30 @ 1
[   11.108415] initcall cros_ec_chardev_driver_init+0x0/0x30 returned 0 after 164 usecs
[   11.116249] calling  cros_ec_lightbar_driver_init+0x0/0x30 @ 1
[   11.122200] initcall cros_ec_lightbar_driver_init+0x0/0x30 returned 0 after 59 usecs
[   11.130040] calling  cros_ec_vbc_driver_init+0x0/0x30 @ 1
[   11.135586] initcall cros_ec_vbc_driver_init+0x0/0x30 returned 0 after 79 usecs
[   11.142961] calling  cros_ec_debugfs_driver_init+0x0/0x30 @ 1
[   11.148832] initcall cros_ec_debugfs_driver_init+0x0/0x30 returned 0 after 80 usecs
[   11.156560] calling  cros_ec_sensorhub_driver_init+0x0/0x30 @ 1
[   11.156635] usb-storage 3-1.3:1.0: USB Mass Storage device detected
[   11.162574] initcall cros_ec_sensorhub_driver_init+0x0/0x30 returned 0 after 47 usecs
[   11.169076] scsi host1: usb-storage 3-1.3:1.0
[   11.176720] calling  cros_ec_sysfs_driver_init+0x0/0x34 @ 1
[   11.186772] initcall cros_ec_sysfs_driver_init+0x0/0x34 returned 0 after 76 usecs
[   11.194316] calling  cros_usbpd_notify_init+0x0/0x50 @ 1
[   11.199768] initcall cros_usbpd_notify_init+0x0/0x50 returned 0 after 98 usecs
[   11.207047] calling  arm_mhu_driver_init+0x0/0x2c @ 1
[   11.212148] initcall arm_mhu_driver_init+0x0/0x2c returned 0 after 17 usecs
[   11.219181] calling  arm_mhu_db_driver_init+0x0/0x2c @ 1
[   11.219193] GPT:Primary header thinks Alt. header is not at the end of the disk.
[   11.224534] initcall arm_mhu_db_driver_init+0x0/0x2c returned 0 after 12 usecs
[   11.231950] GPT:266272 != 30253055
[   11.239203] calling  platform_mhu_driver_init+0x0/0x30 @ 1
[   11.242623] GPT:Alternate GPT header not at the end of the disk.
[   11.248207] initcall platform_mhu_driver_init+0x0/0x30 returned 0 after 74 usecs
[   11.254156] GPT:266272 != 30253055
[   11.254159] GPT: Use GNU Parted to correct GPT errors.
[   11.254175]  mmcblk1: p1
[   11.261593] calling  qcom_glink_ssr_driver_init+0x0/0x34 @ 1
[   11.278385] initcall qcom_glink_ssr_driver_init+0x0/0x34 returned 0 after 12 usecs
[   11.285996] calling  extcon_class_init+0x0/0x38 @ 1
[   11.290909] initcall extcon_class_init+0x0/0x38 returned 0 after 10 usecs
[   11.297729] calling  usb_extcon_driver_init+0x0/0x30 @ 1
[   11.303141] initcall usb_extcon_driver_init+0x0/0x30 returned 0 after 74 usecs
[   11.310400] calling  extcon_cros_ec_driver_init+0x0/0x30 @ 1
[   11.316139] initcall extcon_cros_ec_driver_init+0x0/0x30 returned 0 after 51 usecs
[   11.323747] calling  hisi_l3c_pmu_module_init+0x0/0x98 @ 1
[   11.329291] initcall hisi_l3c_pmu_module_init+0x0/0x98 returned 0 after 31 usecs
[   11.336725] calling  hisi_hha_pmu_module_init+0x0/0x98 @ 1
[   11.342268] initcall hisi_hha_pmu_module_init+0x0/0x98 returned 0 after 31 usecs
[   11.349701] calling  hisi_ddrc_pmu_module_init+0x0/0x98 @ 1
[   11.355331] initcall hisi_ddrc_pmu_module_init+0x0/0x98 returned 0 after 30 usecs
[   11.362850] calling  optee_driver_init+0x0/0x30 @ 1
[   11.367800] initcall optee_driver_init+0x0/0x30 returned 0 after 48 usecs
[   11.374622] calling  alsa_timer_init+0x0/0x23c @ 1
[   11.379532] initcall alsa_timer_init+0x0/0x23c returned 0 after 94 usecs
[   11.386273] calling  alsa_pcm_init+0x0/0x8c @ 1
[   11.390828] initcall alsa_pcm_init+0x0/0x8c returned 0 after 3 usecs
[   11.397211] calling  snd_soc_init+0x0/0x98 @ 1
[   11.401917] initcall snd_soc_init+0x0/0x98 returned 0 after 236 usecs
[   11.408393] calling  wm8904_i2c_driver_init+0x0/0x30 @ 1
[   11.413759] initcall wm8904_i2c_driver_init+0x0/0x30 returned 0 after 28 usecs
[   11.421017] calling  asoc_simple_card_init+0x0/0x30 @ 1
[   11.426330] initcall asoc_simple_card_init+0x0/0x30 returned 0 after 62 usecs
[   11.433500] calling  fsl_sai_driver_init+0x0/0x30 @ 1
[   11.438658] initcall fsl_sai_driver_init+0x0/0x30 returned 0 after 82 usecs
[   11.445653] calling  sock_diag_init+0x0/0x54 @ 1
[   11.450313] initcall sock_diag_init+0x0/0x54 returned 0 after 21 usecs
[   11.456873] calling  init_net_drop_monitor+0x0/0x13c @ 1
[   11.462210] drop_monitor: Initializing network drop monitor service
[   11.468526] initcall init_net_drop_monitor+0x0/0x13c returned 0 after 6168 usecs
[   11.475959] calling  failover_init+0x0/0x34 @ 1
[   11.480511] initcall failover_init+0x0/0x34 returned 0 after 1 usecs
[   11.486899] calling  llc_init+0x0/0x44 @ 1
[   11.491021] initcall llc_init+0x0/0x44 returned 0 after 0 usecs
[   11.496972] calling  snap_init+0x0/0x54 @ 1
[   11.501178] initcall snap_init+0x0/0x54 returned 0 after 2 usecs
[   11.507213] calling  gre_offload_init+0x0/0x68 @ 1
[   11.512029] initcall gre_offload_init+0x0/0x68 returned 0 after 1 usecs
[   11.518676] calling  sysctl_ipv4_init+0x0/0x70 @ 1
[   11.523561] initcall sysctl_ipv4_init+0x0/0x70 returned 0 after 69 usecs
[   11.530297] calling  inet_diag_init+0x0/0xb0 @ 1
[   11.534940] initcall inet_diag_init+0x0/0xb0 returned 0 after 3 usecs
[   11.541412] calling  tcp_diag_init+0x0/0x2c @ 1
[   11.545966] initcall tcp_diag_init+0x0/0x2c returned 0 after 0 usecs
[   11.552350] calling  cubictcp_register+0x0/0x70 @ 1
[   11.557252] initcall cubictcp_register+0x0/0x70 returned 0 after 1 usecs
[   11.563986] calling  inet6_init+0x0/0x350 @ 1
[   11.568584] NET: Registered protocol family 10
[   11.573479] Segment Routing with IPv6
[   11.577204] initcall inet6_init+0x0/0x350 returned 0 after 8631 usecs
[   11.583694] calling  packet_init+0x0/0xa4 @ 1
[   11.588076] NET: Registered protocol family 17
[   11.592544] initcall packet_init+0x0/0xa4 returned 0 after 4364 usecs
[   11.599021] calling  br_init+0x0/0xe4 @ 1
[   11.603067] initcall br_init+0x0/0xe4 returned 0 after 14 usecs
[   11.609018] calling  dsa_init_module+0x0/0xbc @ 1
[   11.613819] initcall dsa_init_module+0x0/0xbc returned 0 after 71 usecs
[   11.620479] calling  dsa_tag_driver_module_init+0x0/0x38 @ 1
[   11.626181] initcall dsa_tag_driver_module_init+0x0/0x38 returned 0 after 6 usecs
[   11.633703] calling  can_init+0x0/0xdc @ 1
[   11.637820] can: controller area network core
[   11.642227] NET: Registered protocol family 29
[   11.646693] initcall can_init+0x0/0xdc returned 0 after 8664 usecs
[   11.652905] calling  raw_module_init+0x0/0x58 @ 1
[   11.657633] can: raw protocol
[   11.660612] initcall raw_module_init+0x0/0x58 returned 0 after 2908 usecs
[   11.667436] calling  bcm_module_init+0x0/0x6c @ 1
[   11.672164] can: broadcast manager protocol
[   11.676368] initcall bcm_module_init+0x0/0x6c returned 0 after 4105 usecs
[   11.683190] calling  cgw_module_init+0x0/0x174 @ 1
[   11.688005] can: netlink gateway - max_hops=1
[   11.692430] initcall cgw_module_init+0x0/0x174 returned 0 after 4321 usecs
[   11.699342] calling  init_rpcsec_gss+0x0/0x90 @ 1
[   11.704087] initcall init_rpcsec_gss+0x0/0x90 returned 0 after 17 usecs
[   11.710736] calling  init_p9+0x0/0x4c @ 1
[   11.714814] 9pnet: Installing 9P2000 support
[   11.719104] initcall init_p9+0x0/0x4c returned 0 after 4237 usecs
[   11.725228] calling  p9_virtio_init+0x0/0x68 @ 1
[   11.729891] initcall p9_virtio_init+0x0/0x68 returned 0 after 18 usecs
[   11.736453] calling  init_dns_resolver+0x0/0x7fac @ 1
[   11.741540] Key type dns_resolver registered
[   11.745827] initcall init_dns_resolver+0x0/0x7fac returned 0 after 4197 usecs
[   11.753057] calling  xen_pm_init+0x0/0x100 @ 1
[   11.757527] initcall xen_pm_init+0x0/0x100 returned -19 after 0 usecs
[   11.764001] calling  init_oops_id+0x0/0x68 @ 1
[   11.768471] initcall init_oops_id+0x0/0x68 returned 0 after 1 usecs
[   11.774770] calling  reboot_ksysfs_init+0x0/0x6c @ 1
[   11.779764] initcall reboot_ksysfs_init+0x0/0x6c returned 0 after 6 usecs
[   11.786587] calling  cpu_latency_qos_init+0x0/0x60 @ 1
[   11.791834] initcall cpu_latency_qos_init+0x0/0x60 returned 0 after 81 usecs
[   11.798923] calling  pm_debugfs_init+0x0/0x44 @ 1
[   11.803659] initcall pm_debugfs_init+0x0/0x44 returned 0 after 8 usecs
[   11.810219] calling  printk_late_init+0x0/0x154 @ 1
[   11.815122] initcall printk_late_init+0x0/0x154 returned 0 after 1 usecs
[   11.821856] calling  init_srcu_module_notifier+0x0/0x50 @ 1
[   11.827457] initcall init_srcu_module_notifier+0x0/0x50 returned 0 after 1 usecs
[   11.834888] calling  swiotlb_create_debugfs+0x0/0x78 @ 1
[   11.840259] initcall swiotlb_create_debugfs+0x0/0x78 returned 0 after 28 usecs
[   11.847523] calling  tk_debug_sleep_time_init+0x0/0x44 @ 1
[   11.853042] initcall tk_debug_sleep_time_init+0x0/0x44 returned 0 after 4 usecs
[   11.860388] calling  taskstats_init+0x0/0x5c @ 1
[   11.865041] registered taskstats version 1
[   11.869154] initcall taskstats_init+0x0/0x5c returned 0 after 4029 usecs
[   11.875888] calling  init_trampolines+0x0/0x34 @ 1
[   11.880706] initcall init_trampolines+0x0/0x34 returned 0 after 1 usecs
[   11.887354] calling  load_system_certificate_list+0x0/0x12c @ 1
[   11.893302] Loading compiled-in X.509 certificates
[   11.898111] initcall load_system_certificate_list+0x0/0x12c returned 0 after 4696 usecs
[   11.906154] calling  fault_around_debugfs+0x0/0x44 @ 1
[   11.911325] initcall fault_around_debugfs+0x0/0x44 returned 0 after 7 usecs
[   11.918321] calling  max_swapfiles_check+0x0/0x18 @ 1
[   11.923398] initcall max_swapfiles_check+0x0/0x18 returned 0 after 0 usecs
[   11.930305] calling  split_huge_pages_debugfs+0x0/0xc0 @ 1
[   11.935821] initcall split_huge_pages_debugfs+0x0/0xc0 returned 0 after 4 usecs
[   11.943167] calling  check_early_ioremap_leak+0x0/0x6c @ 1
[   11.948680] initcall check_early_ioremap_leak+0x0/0x6c returned 0 after 0 usecs
[   11.956024] calling  pstore_init+0x0/0x80 @ 1
[   11.960415] initcall pstore_init+0x0/0x80 returned 0 after 10 usecs
[   11.966716] calling  init_root_keyring+0x0/0x2c @ 1
[   11.971639] initcall init_root_keyring+0x0/0x2c returned 0 after 21 usecs
[   11.978466] calling  integrity_fs_init+0x0/0x78 @ 1
[   11.983376] initcall integrity_fs_init+0x0/0x78 returned 0 after 6 usecs
[   11.990111] calling  blk_timeout_init+0x0/0x24 @ 1
[   11.994924] initcall blk_timeout_init+0x0/0x24 returned 0 after 0 usecs
[   12.001570] calling  prandom_init_late+0x0/0x4c @ 1
[   12.006472] initcall prandom_init_late+0x0/0x4c returned 0 after 0 usecs
[   12.013205] calling  pci_resource_alignment_sysfs_init+0x0/0x38 @ 1
[   12.019509] initcall pci_resource_alignment_sysfs_init+0x0/0x38 returned 0 after 3 usecs
[   12.027642] calling  pci_sysfs_init+0x0/0x70 @ 1
[   12.032285] initcall pci_sysfs_init+0x0/0x70 returned 0 after 1 usecs
[   12.038757] calling  bert_init+0x0/0x254 @ 1
[   12.043048] initcall bert_init+0x0/0x254 returned 0 after 0 usecs
[   12.049172] calling  amba_deferred_retry+0x0/0xb8 @ 1
[   12.054254] initcall amba_deferred_retry+0x0/0xb8 returned 0 after 0 usecs
[   12.061162] calling  clk_debug_init+0x0/0x134 @ 1
[   12.067852] initcall clk_debug_init+0x0/0x134 returned 0 after 1914 usecs
[   12.074689] calling  setup_vcpu_hotplug_event+0x0/0x44 @ 1
[   12.080204] initcall setup_vcpu_hotplug_event+0x0/0x44 returned -19 after 0 usecs
[   12.087731] calling  boot_wait_for_devices+0x0/0x38 @ 1
[   12.092990] initcall boot_wait_for_devices+0x0/0x38 returned 0 after 1 usecs
[   12.100080] calling  sync_state_resume_initcall+0x0/0x28 @ 1
[   12.105773] initcall sync_state_resume_initcall+0x0/0x28 returned 0 after 1 usecs
[   12.113293] calling  deferred_probe_initcall+0x0/0xc0 @ 1
[   12.118853] fsl-edma 22c0000.dma-controller: Adding to iommu group 5
[   12.126408] pci-host-generic 1f0000000.pcie: host bridge /soc/pcie@1f0000000 ranges:
[   12.134221] pci-host-generic 1f0000000.pcie:      MEM 0x01f8000000..0x01f815ffff -> 0x0000000000
[   12.143063] pci-host-generic 1f0000000.pcie:      MEM 0x01f8160000..0x01f81cffff -> 0x0000000000
[   12.151907] pci-host-generic 1f0000000.pcie:      MEM 0x01f81d0000..0x01f81effff -> 0x0000000000
[   12.160748] pci-host-generic 1f0000000.pcie:      MEM 0x01f81f0000..0x01f820ffff -> 0x0000000000
[   12.169586] pci-host-generic 1f0000000.pcie:      MEM 0x01f8210000..0x01f822ffff -> 0x0000000000
[   12.178424] pci-host-generic 1f0000000.pcie:      MEM 0x01f8230000..0x01f824ffff -> 0x0000000000
[   12.187257] pci-host-generic 1f0000000.pcie:      MEM 0x01fc000000..0x01fc3fffff -> 0x0000000000
[   12.196168] pci-host-generic 1f0000000.pcie: ECAM at [mem 0x1f0000000-0x1f00fffff] for [bus 00]
[   12.204987] pci-host-generic 1f0000000.pcie: PCI host bridge to bus 0000:00
[   12.211988] pci_bus 0000:00: root bus resource [bus 00]
[   12.217240] pci_bus 0000:00: root bus resource [mem 0x1f8000000-0x1f815ffff] (bus address [0x00000000-0x0015ffff])
[   12.227637] pci_bus 0000:00: root bus resource [mem 0x1f8160000-0x1f81cffff pref] (bus address [0x00000000-0x0006ffff])
[   12.238473] pci_bus 0000:00: root bus resource [mem 0x1f81d0000-0x1f81effff] (bus address [0x00000000-0x0001ffff])
[   12.248870] pci_bus 0000:00: root bus resource [mem 0x1f81f0000-0x1f820ffff pref] (bus address [0x00000000-0x0001ffff])
[   12.259703] pci_bus 0000:00: root bus resource [mem 0x1f8210000-0x1f822ffff] (bus address [0x00000000-0x0001ffff])
[   12.270103] pci_bus 0000:00: root bus resource [mem 0x1f8230000-0x1f824ffff pref] (bus address [0x00000000-0x0001ffff])
[   12.280938] pci_bus 0000:00: root bus resource [mem 0x1fc000000-0x1fc3fffff] (bus address [0x00000000-0x003fffff])
[   12.291351] pci 0000:00:00.0: [1957:e100] type 00 class 0x020001
[   12.297415] pci 0000:00:00.0: BAR 0: [mem 0x1f8000000-0x1f803ffff 64bit] (from Enhanced Allocation, properties 0x0)
[   12.307902] pci 0000:00:00.0: BAR 2: [mem 0x1f8160000-0x1f816ffff 64bit pref] (from Enhanced Allocation, properties 0x1)
[   12.318825] pci 0000:00:00.0: VF BAR 0: [mem 0x1f81d0000-0x1f81dffff 64bit] (from Enhanced Allocation, properties 0x4)
[   12.329573] pci 0000:00:00.0: VF BAR 2: [mem 0x1f81f0000-0x1f81fffff 64bit pref] (from Enhanced Allocation, properties 0x3)
[   12.340775] pci 0000:00:00.0: PME# supported from D0 D3hot
[   12.346294] pci 0000:00:00.0: VF(n) BAR0 space: [mem 0x1f81d0000-0x1f81effff 64bit] (contains BAR0 for 2 VFs)
[   12.356257] pci 0000:00:00.0: VF(n) BAR2 space: [mem 0x1f81f0000-0x1f820ffff 64bit pref] (contains BAR2 for 2 VFs)
[   12.366799] pci 0000:00:00.1: [1957:e100] type 00 class 0x020001
[   12.372858] pci 0000:00:00.1: BAR 0: [mem 0x1f8040000-0x1f807ffff 64bit] (from Enhanced Allocation, properties 0x0)
[   12.383346] pci 0000:00:00.1: BAR 2: [mem 0x1f8170000-0x1f817ffff 64bit pref] (from Enhanced Allocation, properties 0x1)
[   12.394267] pci 0000:00:00.1: VF BAR 0: [mem 0x1f8210000-0x1f821ffff 64bit] (from Enhanced Allocation, properties 0x4)
[   12.405018] pci 0000:00:00.1: VF BAR 2: [mem 0x1f8230000-0x1f823ffff 64bit pref] (from Enhanced Allocation, properties 0x3)
[   12.416216] pci 0000:00:00.1: PME# supported from D0 D3hot
[   12.421733] pci 0000:00:00.1: VF(n) BAR0 space: [mem 0x1f8210000-0x1f822ffff 64bit] (contains BAR0 for 2 VFs)
[   12.431695] pci 0000:00:00.1: VF(n) BAR2 space: [mem 0x1f8230000-0x1f824ffff 64bit pref] (contains BAR2 for 2 VFs)
[   12.442227] pci 0000:00:00.1: Linked as a sync state only consumer to 0000:00:00.1
[   12.449858] pci 0000:00:00.2: [1957:e100] type 00 class 0x020001
[   12.455914] pci 0000:00:00.2: BAR 0: [mem 0x1f8080000-0x1f80bffff 64bit] (from Enhanced Allocation, properties 0x0)
[   12.466404] pci 0000:00:00.2: BAR 2: [mem 0x1f8180000-0x1f818ffff 64bit pref] (from Enhanced Allocation, properties 0x1)
[   12.477339] pci 0000:00:00.2: PME# supported from D0 D3hot
[   12.482959] pci 0000:00:00.3: [1957:ee01] type 00 class 0x088001
[   12.489020] pci 0000:00:00.3: BAR 0: [mem 0x1f8100000-0x1f811ffff 64bit] (from Enhanced Allocation, properties 0x0)
[   12.499507] pci 0000:00:00.3: BAR 2: [mem 0x1f8190000-0x1f819ffff 64bit pref] (from Enhanced Allocation, properties 0x1)
[   12.510440] pci 0000:00:00.3: PME# supported from D0 D3hot
[   12.516061] pci 0000:00:00.4: [1957:ee02] type 00 class 0x088001
[   12.522119] pci 0000:00:00.4: BAR 0: [mem 0x1f8120000-0x1f813ffff 64bit] (from Enhanced Allocation, properties 0x0)
[   12.532608] pci 0000:00:00.4: BAR 2: [mem 0x1f81a0000-0x1f81affff 64bit pref] (from Enhanced Allocation, properties 0x1)
[   12.543542] pci 0000:00:00.4: PME# supported from D0 D3hot
[   12.549155] pci 0000:00:00.5: [1957:eef0] type 00 class 0x020801
[   12.555212] pci 0000:00:00.5: BAR 0: [mem 0x1f8140000-0x1f815ffff 64bit] (from Enhanced Allocation, properties 0x0)
[   12.565699] pci 0000:00:00.5: BAR 2: [mem 0x1f81b0000-0x1f81bffff 64bit pref] (from Enhanced Allocation, properties 0x1)
[   12.576644] pci 0000:00:00.5: BAR 4: [mem 0x1fc000000-0x1fc3fffff 64bit] (from Enhanced Allocation, properties 0x0)
[   12.587173] pci 0000:00:00.5: PME# supported from D0 D3hot
[   12.587188] scsi 1:0:0:0: Direct-Access     JetFlash Transcend 32GB   1100 PQ: 0 ANSI: 6
[   12.592817] pci 0000:00:00.6: [1957:e100] type 00 class 0x020001
[   12.606895] pci 0000:00:00.6: BAR 0: [mem 0x1f80c0000-0x1f80fffff 64bit] (from Enhanced Allocation, properties 0x0)
[   12.617393] pci 0000:00:00.6: BAR 2: [mem 0x1f81c0000-0x1f81cffff 64bit pref] (from Enhanced Allocation, properties 0x1)
[   12.628345] pci 0000:00:00.6: PME# supported from D0 D3hot
[   12.628382] sd 1:0:0:0: [sda] 61702144 512-byte logical blocks: (31.6 GB/29.4 GiB)
[   12.634798] pci 0000:00:1f.0: [1957:e001] type 00 class 0x080700
[   12.647590] OF: /soc/pcie@1f0000000: no msi-map translation for id 0xf8 on (null)
[   12.655149] sd 1:0:0:0: [sda] Write Protect is off
[   12.655295] pci 0000:00:00.0: calling  quirk_fsl_no_msi+0x0/0x30 @ 89
[   12.659969] sd 1:0:0:0: [sda] Mode Sense: 43 00 00 00
[   12.666433] pci 0000:00:00.0: quirk_fsl_no_msi+0x0/0x30 took 0 usecs
[   12.678023] fsl_enetc 0000:00:00.0: Linked as a consumer to 5000000.iommu
[   12.684875] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   12.684884] fsl_enetc 0000:00:00.0: Adding to iommu group 6
[   12.701791]  sda: sda1
[   12.706485] sd 1:0:0:0: [sda] Attached SCSI removable disk
[   12.805947] fsl_enetc 0000:00:00.0: enabling device (0400 -> 0402)
[   12.812200] fsl_enetc 0000:00:00.0: no MAC address specified for SI1, using 8e:d8:2f:4a:a1:8e
[   12.820769] fsl_enetc 0000:00:00.0: no MAC address specified for SI2, using c6:34:53:33:70:32
[   12.829753] libphy: Freescale ENETC MDIO Bus: probed
[   12.835987] libphy: Freescale ENETC internal MDIO Bus: probed
[   12.842300] pci 0000:00:00.1: calling  quirk_fsl_no_msi+0x0/0x30 @ 89
[   12.848791] pci 0000:00:00.1: quirk_fsl_no_msi+0x0/0x30 took 0 usecs
[   12.855291] fsl_enetc 0000:00:00.1: Linked as a consumer to 5000000.iommu
[   12.862151] fsl_enetc 0000:00:00.1: Adding to iommu group 7
[   12.973945] fsl_enetc 0000:00:00.1: enabling device (0400 -> 0402)
[   12.980198] fsl_enetc 0000:00:00.1: no MAC address specified for SI0, using 12:92:e7:84:ab:1c
[   12.988766] fsl_enetc 0000:00:00.1: no MAC address specified for SI1, using f2:4c:fa:ab:57:02
[   12.997332] fsl_enetc 0000:00:00.1: no MAC address specified for SI2, using c6:14:47:66:62:2d
[   13.006327] mdio_bus 0000:00:00.1: Linked as a sync state only consumer to 0000:00:00.1
[   13.014402] libphy: Freescale ENETC MDIO Bus: probed
[   13.019875] mdio_bus 0000:00:00.1:04: Linked as a consumer to 0000:00:00.1
[   13.026817] mdio_bus 0000:00:00.1:04: probe deferral - supplier 0000:00:00.1 not ready
[   13.035311] pci 0000:00:00.2: calling  quirk_fsl_no_msi+0x0/0x30 @ 89
[   13.041806] pci 0000:00:00.2: quirk_fsl_no_msi+0x0/0x30 took 0 usecs
[   13.048295] fsl_enetc 0000:00:00.2: Linked as a consumer to 5000000.iommu
[   13.055146] fsl_enetc 0000:00:00.2: Adding to iommu group 8
[   13.165945] fsl_enetc 0000:00:00.2: enabling device (0400 -> 0402)
[   13.172192] fsl_enetc 0000:00:00.2: no MAC address specified for SI0, using 1e:9a:57:30:a7:52
[   13.181494] pci 0000:00:00.3: calling  quirk_fsl_no_msi+0x0/0x30 @ 89
[   13.188006] pci 0000:00:00.3: quirk_fsl_no_msi+0x0/0x30 took 0 usecs
[   13.194500] fsl_enetc_mdio 0000:00:00.3: Linked as a consumer to 5000000.iommu
[   13.201786] fsl_enetc_mdio 0000:00:00.3: Adding to iommu group 9
[   13.313948] fsl_enetc_mdio 0000:00:00.3: enabling device (0400 -> 0402)
[   13.320784] libphy: FSL PCIe IE Central MDIO Bus: probed
[   13.326323] mdio_bus 0000:00:00.3: MDIO device at address 7 is missing.
[   13.333163] mdio_bus 0000:00:00.3: MDIO device at address 8 is missing.
[   13.340003] mdio_bus 0000:00:00.3: MDIO device at address 9 is missing.
[   13.346843] mdio_bus 0000:00:00.3: MDIO device at address 10 is missing.
[   13.353605] pci 0000:00:00.4: calling  quirk_fsl_no_msi+0x0/0x30 @ 89
[   13.360085] pci 0000:00:00.4: quirk_fsl_no_msi+0x0/0x30 took 0 usecs
[   13.366516] pci 0000:00:00.5: calling  quirk_fsl_no_msi+0x0/0x30 @ 89
[   13.372992] pci 0000:00:00.5: quirk_fsl_no_msi+0x0/0x30 took 0 usecs
[   13.379458] mscc_felix 0000:00:00.5: Linked as a consumer to 5000000.iommu
[   13.386387] mscc_felix 0000:00:00.5: Adding to iommu group 10
[   13.392299] mscc_felix 0000:00:00.5: enabling device (0404 -> 0406)
[   13.405882] libphy: VSC9959 internal MDIO bus: probed
[   13.410978] mscc_felix 0000:00:00.5: Found PCS at internal MDIO address 0
[   13.417801] mscc_felix 0000:00:00.5: Found PCS at internal MDIO address 1
[   13.424626] mscc_felix 0000:00:00.5: Found PCS at internal MDIO address 2
[   13.431447] mscc_felix 0000:00:00.5: Found PCS at internal MDIO address 3
[   13.453501] mscc_felix 0000:00:00.5 swp0 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 0
[   13.463722] mscc_felix 0000:00:00.5 swp1 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 1
[   13.473944] mscc_felix 0000:00:00.5 swp2 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 2
[   13.484136] mscc_felix 0000:00:00.5 swp3 (uninitialized): error -19 setting up PHY for tree 0, switch 0, port 3
[   13.494329] mscc_felix 0000:00:00.5: configuring for fixed/internal link mode
[   13.501527] mscc_felix 0000:00:00.5: Link is Up - 2.5Gbps/Full - flow control off
[   13.509099] mscc_felix 0000:00:00.5: Linked as a consumer to 0000:00:00.2
[   13.515924] device eth2 entered promiscuous mode
[   13.520585] DSA: tree 0 setup
[   13.523614] pci 0000:00:00.6: calling  quirk_fsl_no_msi+0x0/0x30 @ 89
[   13.530097] pci 0000:00:00.6: quirk_fsl_no_msi+0x0/0x30 took 0 usecs
[   13.536586] fsl_enetc 0000:00:00.6: Linked as a consumer to 5000000.iommu
[   13.543434] fsl_enetc 0000:00:00.6: Adding to iommu group 11
[   13.549232] fsl_enetc 0000:00:00.6: device is disabled, skipping
[   13.555296] pci 0000:00:1f.0: calling  quirk_fsl_no_msi+0x0/0x30 @ 89
[   13.561773] pci 0000:00:1f.0: quirk_fsl_no_msi+0x0/0x30 took 0 usecs
[   13.568177] OF: /soc/pcie@1f0000000: no iommu-map translation for id 0xf8 on (null)
[   13.576001] pcieport 0000:00:1f.0: PME: Signaling with IRQ 129
[   13.582037] pcieport 0000:00:1f.0: AER: enabled with IRQ 129
[   13.588437] 2270000.serial: ttyLP2 at MMIO 0x2270000 (irq = 25, base_baud = 12500000) is a FSL_LPUART
[   13.598727] VDDIO: Bringing 1500000uV into 1800000-1800000uV
[   13.605716] initcall deferred_probe_initcall+0x0/0xc0 returned 0 after 1452145 usecs
[   13.613523] calling  genpd_power_off_unused+0x0/0xa0 @ 1
[   13.618868] initcall genpd_power_off_unused+0x0/0xa0 returned 0 after 0 usecs
[   13.626039] calling  genpd_debug_init+0x0/0x98 @ 1
[   13.630862] initcall genpd_debug_init+0x0/0x98 returned 0 after 8 usecs
[   13.637510] calling  gpio_keys_init+0x0/0x30 @ 1
[   13.644616] input: buttons0 as /devices/platform/buttons0/input/input1
[   13.651361] initcall gpio_keys_init+0x0/0x30 returned 0 after 8991 usecs
[   13.658120] calling  register_update_efi_random_seed+0x0/0x44 @ 1
[   13.664249] initcall register_update_efi_random_seed+0x0/0x44 returned 0 after 0 usecs
[   13.672215] calling  efi_shutdown_init+0x0/0x68 @ 1
[   13.677119] initcall efi_shutdown_init+0x0/0x68 returned -19 after 0 usecs
[   13.684028] calling  efi_earlycon_unmap_fb+0x0/0x4c @ 1
[   13.689281] initcall efi_earlycon_unmap_fb+0x0/0x4c returned 0 after 0 usecs
[   13.696364] calling  of_fdt_raw_init+0x0/0x94 @ 1
[   13.701145] initcall of_fdt_raw_init+0x0/0x94 returned 0 after 50 usecs
[   13.707797] calling  tcp_congestion_default+0x0/0x34 @ 1
[   13.713138] initcall tcp_congestion_default+0x0/0x34 returned 0 after 1 usecs
[   13.720308] calling  ip_auto_config+0x0/0xea8 @ 1
[   13.725050] initcall ip_auto_config+0x0/0xea8 returned 0 after 12 usecs
[   13.731698] calling  init_amu_fie+0x0/0x29c @ 1
[   13.736257] initcall init_amu_fie+0x0/0x29c returned 0 after 1 usecs
[   13.742641] calling  software_resume+0x0/0x1c0 @ 1
[   13.747462] initcall software_resume+0x0/0x1c0 returned -2 after 1 usecs
[   13.754195] calling  trace_eval_sync+0x0/0x34 @ 1
[   13.758938] initcall trace_eval_sync+0x0/0x34 returned 0 after 11 usecs
[   13.765592] calling  clear_boot_tracer+0x0/0x44 @ 1
[   13.770500] initcall clear_boot_tracer+0x0/0x44 returned 0 after 0 usecs
[   13.777236] calling  acpi_gpio_handle_deferred_request_irqs+0x0/0xb4 @ 1
[   13.783976] initcall acpi_gpio_handle_deferred_request_irqs+0x0/0xb4 returned 0 after 0 usecs
[   13.792543] calling  fb_logo_late_init+0x0/0x24 @ 1
[   13.797442] initcall fb_logo_late_init+0x0/0x24 returned 0 after 0 usecs
[   13.804173] calling  clk_disable_unused+0x0/0xe0 @ 1
[   13.809185] initcall clk_disable_unused+0x0/0xe0 returned 0 after 21 usecs
[   13.816095] calling  regulator_init_complete+0x0/0x5c @ 1
[   13.821519] initcall regulator_init_complete+0x0/0x5c returned 0 after 1 usecs
[   13.828792] calling  of_platform_sync_state_init+0x0/0x28 @ 1
[   13.834577] initcall of_platform_sync_state_init+0x0/0x28 returned 0 after 0 usecs
[   13.842190] calling  alsa_sound_last_init+0x0/0x98 @ 1
[   13.847355] ALSA device list:
[   13.850335]   No soundcards found.
[   13.853744] initcall alsa_sound_last_init+0x0/0x98 returned 0 after 6239 usecs
[   13.864983] EXT4-fs (mmcblk0p2): INFO: recovery required on readonly filesystem
[   13.872341] EXT4-fs (mmcblk0p2): write access will be enabled during recovery
[   13.885041] EXT4-fs (mmcblk0p2): recovery complete
[   13.891415] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[   13.901267] VFS: Mounted root (ext4 filesystem) readonly on device 179:34.
[   13.908420] devtmpfs: mounted
[   13.916116] Freeing unused kernel memory: 4800K
[   13.925996] Missing param value! Expected 'debug=...value...'
[   13.931775] Missing param value! Expected 'rootwait=...value...'
[   13.937808] Missing param value! Expected 'initcall_debug=...value...'
[   13.944365] Run /sbin/init as init process
[   13.948480]   with arguments:
[   13.951457]     /sbin/init
[   13.954174]   with environment:
[   13.957321]     HOME=/
[   13.959689]     TERM=linux
[   13.990473] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null). Quota mode: none.
#23 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 36 lines · Michael Walle

Am 2021-01-18 22:01, schrieb Saravana Kannan:

Quoted 34 lines · Michael Walle
Quoted 32 lines
Quoted 20 lines

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

This breaks (at least) probing of the PCIe controllers of my board. The
driver in question is
drivers/pci/controller/dwc/pci-layerscape.c
I've also put the maintainers of this driver on CC. Looks like it uses a
proper struct device. But it uses builtin_platform_driver_probe() and apparently it waits for the iommu which uses module_platform_driver(). Dunno if that will work together.

Quoted 6 lines
Quoted 4 lines

Yeah, the builtin vs module doesn't matter. I've had fw_devlink work multiple times with the consumer driver being built in and the supplier actually loaded as a module. Making that work is one of the goals of fw_devlink.

Ok.

Hi Michael,

My bad, I spoke too soon. I thought you were talking about builtin_ vs module_. My response is correct in that context. But the problem here is related to builtin_platform_driver_probe(). That macro expects the device (PCI) to be added and ready to probe by the time it's called. If not, it just gives up and frees the code. That's why it's not getting called after the first attempt. Can you please convert it into builtin_platform_driver()? It should be a pretty trivial change.

-Saravana

Quoted 40 lines
Quoted 13 lines
Quoted 5 lines

The board device tree can be found here:
arch/arm64/boot/dts/freescale/fsl-ls1028a-kontron-sl28-var3-ads2.dts

Attached is the log with enabled "probe deferral" messages enabled.

I took a look at the logs. As you said, pci seems to be waiting on iommu, but it's not clear why the iommu didn't probe by then. Can you add initcall_debug=1 and enable the logs in device_link_add()? Btw, I realize one compromise on the logs is to send them as an attachment instead of inline. That way, it's still archived in the list, but I don't have to deal with log lines getting wrapped, etc.

Quoted 7 lines

Thanks for reporting the issues. Also, could you try picking up all of these changes and giving it a shot. It's unlikely to help, but I want to rule out issues related to fixes in progress.

https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

Did pick them up, the last one had a conflict due some superfluous lines.
Maybe they got reordered in that arrray.

Issue still persist. I've enabled the debug in device_link_add(), in device_links_check_suppliers() and booted with initcall_debug. Please see attached log. Lets see how that goes ;)

[    0.132687] calling  ls_pcie_driver_init+0x0/0x38 @ 1
[    0.132762] platform 3400000.pcie: probe deferral - supplier
5000000.iommu not ready
[    0.132777] platform 3500000.pcie: probe deferral - supplier
5000000.iommu not ready
[    0.132818] initcall ls_pcie_driver_init+0x0/0x38 returned -19 after

119 usecs

After that, ls_pcie_driver_init() is never called again.

-michael

#24 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

In reply to Saravana Kannan

Hi Saravana,

Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#25 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Hi Geert,

Quoted 26 lines · Geert Uytterhoeven

Hi Saravana,

Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

There is a tentative patch from Saravana here[1], which works around some issues on my RK3399 platform, and it'd be interesting to find out whether that helps on your system.

Thanks,

        M.

[1] https://lore.kernel.org/r/[email protected]

Signature · 1 line
Jazz is not dead. It just smells funny...
#26 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Hi Marc,

Quoted 40 lines · Marc Zyngier
Quoted 28 lines · Geert Uytterhoeven
Quoted 22 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently,
only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their
status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

There is a tentative patch from Saravana here[1], which works around some issues on my RK3399 platform, and it'd be interesting to find out whether that helps on your system.

Quoted 6 lines

Thanks,

         M.

[1]
https://lore.kernel.org/r/[email protected]

Thanks for the suggestion, but given no devices probe (incl. GPIO providers), I'm afraid it won't help. [testing] Indeed.

With the debug prints in device_links_check_suppliers enabled, and some postprocessing, I get:

255 supplier e6180000.system-controller not ready
9 supplier fe990000.iommu not ready
9 supplier fe980000.iommu not ready
6 supplier febd0000.iommu not ready
6 supplier ec670000.iommu not ready
3 supplier febe0000.iommu not ready
3 supplier e7740000.iommu not ready
3 supplier e6740000.iommu not ready
3 supplier e65ee000.usb-phy not ready
3 supplier e6570000.iommu not ready
3 supplier e6054000.gpio not ready
3 supplier e6053000.gpio not ready

As everything is part of a PM Domain, the (lack of the) system controller must be the culprit. What's wrong with it? It is registered very early in the boot:

[    0.142096] rcar_sysc_pd_init:442: of_genpd_add_provider_onecell() returned 0

Thanks!

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#27 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 69 lines · Geert Uytterhoeven

Hi Marc,

Quoted 40 lines · Marc Zyngier
Quoted 28 lines · Geert Uytterhoeven
Quoted 22 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently,
only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their
status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

There is a tentative patch from Saravana here[1], which works around some issues on my RK3399 platform, and it'd be interesting to find out whether that helps on your system.

Quoted 6 lines

Thanks,

         M.

[1]
https://lore.kernel.org/r/[email protected]

Thanks for the suggestion, but given no devices probe (incl. GPIO providers), I'm afraid it won't help. [testing] Indeed.

With the debug prints in device_links_check_suppliers enabled, and some postprocessing, I get:

255 supplier e6180000.system-controller not ready
9 supplier fe990000.iommu not ready
9 supplier fe980000.iommu not ready
6 supplier febd0000.iommu not ready
6 supplier ec670000.iommu not ready
3 supplier febe0000.iommu not ready
3 supplier e7740000.iommu not ready
3 supplier e6740000.iommu not ready
3 supplier e65ee000.usb-phy not ready
3 supplier e6570000.iommu not ready
3 supplier e6054000.gpio not ready
3 supplier e6053000.gpio not ready

Quoted 5 lines

As everything is part of a PM Domain, the (lack of the) system controller must be the culprit. What's wrong with it? It is registered very early in the boot:

[    0.142096] rcar_sysc_pd_init:442: of_genpd_add_provider_onecell() returned 0

Yeah, this looks like the exact same problem. The devlink stuff assumes that because there is a "compatible" property, there will be a driver directly associated with the node containing this property.

If any other node has a reference to that first node, the dependency will only get resolved if/when that first node is bound to a driver. Trouble is, there are *tons* of code in the tree that invalidate this heuristic, and for each occurrence of this we get another failure.

The patch I referred to papers over it by registering a dummy driver, but that doesn't scale easily...

        M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#28 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

In reply to Geert Uytterhoeven
Quoted 70 lines · Geert Uytterhoeven

Hi Marc,

Quoted 40 lines · Marc Zyngier
Quoted 28 lines · Geert Uytterhoeven
Quoted 22 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently,
only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their
status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

There is a tentative patch from Saravana here[1], which works around some issues on my RK3399 platform, and it'd be interesting to find out whether that helps on your system.

Quoted 6 lines

Thanks,

         M.

[1]
https://lore.kernel.org/r/[email protected]

Thanks for the suggestion, but given no devices probe (incl. GPIO providers), I'm afraid it won't help. [testing] Indeed.

With the debug prints in device_links_check_suppliers enabled, and some postprocessing, I get:

255 supplier e6180000.system-controller not ready
9 supplier fe990000.iommu not ready
9 supplier fe980000.iommu not ready
6 supplier febd0000.iommu not ready
6 supplier ec670000.iommu not ready
3 supplier febe0000.iommu not ready
3 supplier e7740000.iommu not ready
3 supplier e6740000.iommu not ready
3 supplier e65ee000.usb-phy not ready
3 supplier e6570000.iommu not ready
3 supplier e6054000.gpio not ready
3 supplier e6053000.gpio not ready

Quoted 5 lines

As everything is part of a PM Domain, the (lack of the) system controller must be the culprit. What's wrong with it? It is registered very early in the boot:

[    0.142096] rcar_sysc_pd_init:442: of_genpd_add_provider_onecell() returned 0

Hi Geert,

Thanks for reporting the issue.

Looks like you found the important logs. Can you please enable all these logs and send the early con logs as an attachment (so I don't need to deal with lines getting wrapped)?
1. The ones in device_links_check_suppliers()
2. The ones in device_link_add()
3. initcall_debug=1

That should help us figure out what's going on. Also, what's the DT that corresponds to one of the boards that see this issue?

Lastly, can you please pick up these 3 patches (some need clean up before they merge) to make sure it's not an issue being worked on from other bug reports?
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

I have a strong hunch the 2nd one will fix your issues. fw_devlink can handle cyclic dependencies now (it basically reverts to fw_devlink=permissive mode for devices in the cycle), but it needs to "see" all the dependencies to know there's a cycle. So want to make sure it "sees" the "gpios" binding used all over some of the Renesas DT files.

Thanks,
Saravana

#29 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Hi Saravana,

Quoted 73 lines · Saravana Kannan
Quoted 71 lines · Geert Uytterhoeven
Quoted 40 lines · Marc Zyngier
Quoted 28 lines · Geert Uytterhoeven
Quoted 22 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently,
only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their
status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

There is a tentative patch from Saravana here[1], which works around some issues on my RK3399 platform, and it'd be interesting to find out whether that helps on your system.

Quoted 6 lines

Thanks,

         M.

[1]
https://lore.kernel.org/r/[email protected]

Thanks for the suggestion, but given no devices probe (incl. GPIO providers), I'm afraid it won't help. [testing] Indeed.

With the debug prints in device_links_check_suppliers enabled, and some postprocessing, I get:

255 supplier e6180000.system-controller not ready
9 supplier fe990000.iommu not ready
9 supplier fe980000.iommu not ready
6 supplier febd0000.iommu not ready
6 supplier ec670000.iommu not ready
3 supplier febe0000.iommu not ready
3 supplier e7740000.iommu not ready
3 supplier e6740000.iommu not ready
3 supplier e65ee000.usb-phy not ready
3 supplier e6570000.iommu not ready
3 supplier e6054000.gpio not ready
3 supplier e6053000.gpio not ready

As everything is part of a PM Domain, the (lack of the) system controller must be the culprit. What's wrong with it? It is registered very early in the boot:

Quoted 8 lines
Quoted 1 line
[    0.142096] rcar_sysc_pd_init:442: of_genpd_add_provider_onecell() returned 0

Looks like you found the important logs. Can you please enable all these logs and send the early con logs as an attachment (so I don't need to deal with lines getting wrapped)?
1. The ones in device_links_check_suppliers()
2. The ones in device_link_add()
3. initcall_debug=1

I have attached[*] the requested log.

Quoted 2 lines

That should help us figure out what's going on. Also, what's the DT that corresponds to one of the boards that see this issue?

arch/arm64/boot/dts/renesas/r8a77951-salvator-xs.dts

Quoted 7 lines

Lastly, can you please pick up these 3 patches (some need clean up before they merge) to make sure it's not an issue being worked on from other bug reports?
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

I have a strong hunch the 2nd one will fix your issues. fw_devlink can handle cyclic dependencies now (it basically reverts to fw_devlink=permissive mode for devices in the cycle), but it needs to "see" all the dependencies to know there's a cycle. So want to make sure it "sees" the "gpios" binding used all over some of the Renesas DT files.

These patches don't help.
The 2nd one actually introduces a new failure:

     OF: /soc/i2c@e66d8000/gpio@20/pcie-sata-switch-hog: could not get

#gpio-cells for /cpus/cpu@102

Note that my issues don't seem to be GPIO-related at all.

BTW, you are aware IOMMUs and DMA controllers are optional? I.e. device drivers with iommus and/or dmas DT properties where the targets of these properties do not have a driver should still be probed, eventually. But if the IOMMU or DMA drivers are present, they should be probed first, so the device drivers can make use of them.

Thanks!

[*] Although attaching means people like myself cannot read and comment
on the log easily, without saving the attachment first. That's also the reason why patches should be submitted inline...

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#30 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 113 lines · Geert Uytterhoeven

Hi Saravana,

Quoted 73 lines · Saravana Kannan
Quoted 71 lines · Geert Uytterhoeven
Quoted 40 lines · Marc Zyngier
Quoted 28 lines · Geert Uytterhoeven
Quoted 22 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently,
only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their
status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

There is a tentative patch from Saravana here[1], which works around some issues on my RK3399 platform, and it'd be interesting to find out whether that helps on your system.

Quoted 6 lines

Thanks,

         M.

[1]
https://lore.kernel.org/r/[email protected]

Thanks for the suggestion, but given no devices probe (incl. GPIO providers), I'm afraid it won't help. [testing] Indeed.

With the debug prints in device_links_check_suppliers enabled, and some postprocessing, I get:

255 supplier e6180000.system-controller not ready
9 supplier fe990000.iommu not ready
9 supplier fe980000.iommu not ready
6 supplier febd0000.iommu not ready
6 supplier ec670000.iommu not ready
3 supplier febe0000.iommu not ready
3 supplier e7740000.iommu not ready
3 supplier e6740000.iommu not ready
3 supplier e65ee000.usb-phy not ready
3 supplier e6570000.iommu not ready
3 supplier e6054000.gpio not ready
3 supplier e6053000.gpio not ready

As everything is part of a PM Domain, the (lack of the) system controller must be the culprit. What's wrong with it? It is registered very early in the boot:

Quoted 8 lines
Quoted 1 line
[    0.142096] rcar_sysc_pd_init:442: of_genpd_add_provider_onecell() returned 0

Looks like you found the important logs. Can you please enable all these logs and send the early con logs as an attachment (so I don't need to deal with lines getting wrapped)?
1. The ones in device_links_check_suppliers()
2. The ones in device_link_add()
3. initcall_debug=1

I have attached[*] the requested log.

Quoted 2 lines

That should help us figure out what's going on. Also, what's the DT that corresponds to one of the boards that see this issue?

arch/arm64/boot/dts/renesas/r8a77951-salvator-xs.dts

Quoted 7 lines

Lastly, can you please pick up these 3 patches (some need clean up before they merge) to make sure it's not an issue being worked on from other bug reports?
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

I have a strong hunch the 2nd one will fix your issues. fw_devlink can handle cyclic dependencies now (it basically reverts to fw_devlink=permissive mode for devices in the cycle), but it needs to "see" all the dependencies to know there's a cycle. So want to make sure it "sees" the "gpios" binding used all over some of the Renesas DT files.

These patches don't help.
The 2nd one actually introduces a new failure:

     OF: /soc/i2c@e66d8000/gpio@20/pcie-sata-switch-hog: could not get

#gpio-cells for /cpus/cpu@102

Quoted 7 lines

Note that my issues don't seem to be GPIO-related at all.

BTW, you are aware IOMMUs and DMA controllers are optional? I.e. device drivers with iommus and/or dmas DT properties where the targets of these properties do not have a driver should still be probed, eventually. But if the IOMMU or DMA drivers are present, they should be probed first, so the device drivers can make use of them.

Thanks for the logs and details.

Yeah, this is going to be a problem then. How is this handled in static kernels today? Do we just try to make sure the iommus driver probes the iommu device before the consumers? And then the consumers simply don't defer probe on failure to get iommu?

I can make this work if modules are not enabled (needs some code changes), but it's not going to work when there are modules. There's no way to tell if an iommu module won't be loaded soon. Also, device links doing this behavior only for iommu/dma is probably not a good idea. So, whatever we do will have to be common behavior. :(

Another intermediate option I was thinking was having a CONFIG_FW_DEVLINK_OFF/PERMISSIVE/ON and defaulting it to ON for ARM64 and turning it off in the defconfig for boards for which this doesn't work. That way, we can incrementally enable fw_devlink.

This week is a very hectic week for me. So, please bear with slow responses from me for rest of this week. Let me think about this a bit to see if I can come up with a better solution than what I have in mind.

Also, can you try deleting "iommu" and "dma" parsing in of_supplier_bindings[] in driver/of/property.c and see if it helps? Then we'd know this is the reason for things not working in your case.

Quoted 5 lines

Thanks!

[*] Although attaching means people like myself cannot read and comment
on the log easily, without saving the attachment first. That's also the reason why patches should be submitted inline...

Yeah, I see your concern. If you want to add comments to logs when sending them, yeah, please go ahead and put it inline. Or if someone wants to add comments to what you attached, they could copy paste the relevant sections and add comments.

Thanks,
Saravana

#31 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 141 lines · Saravana Kannan
Quoted 113 lines · Geert Uytterhoeven

Hi Saravana,

Quoted 73 lines · Saravana Kannan
Quoted 71 lines · Geert Uytterhoeven
Quoted 40 lines · Marc Zyngier
Quoted 28 lines · Geert Uytterhoeven
Quoted 22 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently,
only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their
status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

There is a tentative patch from Saravana here[1], which works around some issues on my RK3399 platform, and it'd be interesting to find out whether that helps on your system.

Quoted 6 lines

Thanks,

         M.

[1]
https://lore.kernel.org/r/[email protected]

Thanks for the suggestion, but given no devices probe (incl. GPIO providers), I'm afraid it won't help. [testing] Indeed.

With the debug prints in device_links_check_suppliers enabled, and some postprocessing, I get:

255 supplier e6180000.system-controller not ready
9 supplier fe990000.iommu not ready
9 supplier fe980000.iommu not ready
6 supplier febd0000.iommu not ready
6 supplier ec670000.iommu not ready
3 supplier febe0000.iommu not ready
3 supplier e7740000.iommu not ready
3 supplier e6740000.iommu not ready
3 supplier e65ee000.usb-phy not ready
3 supplier e6570000.iommu not ready
3 supplier e6054000.gpio not ready
3 supplier e6053000.gpio not ready

As everything is part of a PM Domain, the (lack of the) system controller must be the culprit. What's wrong with it? It is registered very early in the boot:

Quoted 8 lines
Quoted 1 line
[    0.142096] rcar_sysc_pd_init:442: of_genpd_add_provider_onecell() returned 0

Looks like you found the important logs. Can you please enable all these logs and send the early con logs as an attachment (so I don't need to deal with lines getting wrapped)?
1. The ones in device_links_check_suppliers()
2. The ones in device_link_add()
3. initcall_debug=1

I have attached[*] the requested log.

Quoted 2 lines

That should help us figure out what's going on. Also, what's the DT that corresponds to one of the boards that see this issue?

arch/arm64/boot/dts/renesas/r8a77951-salvator-xs.dts

Quoted 7 lines

Lastly, can you please pick up these 3 patches (some need clean up before they merge) to make sure it's not an issue being worked on from other bug reports?
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

Quoted 6 lines

I have a strong hunch the 2nd one will fix your issues. fw_devlink can handle cyclic dependencies now (it basically reverts to fw_devlink=permissive mode for devices in the cycle), but it needs to "see" all the dependencies to know there's a cycle. So want to make sure it "sees" the "gpios" binding used all over some of the Renesas DT files.

These patches don't help.
The 2nd one actually introduces a new failure:

     OF: /soc/i2c@e66d8000/gpio@20/pcie-sata-switch-hog: could not get

#gpio-cells for /cpus/cpu@102

Quoted 7 lines

Note that my issues don't seem to be GPIO-related at all.

BTW, you are aware IOMMUs and DMA controllers are optional? I.e. device drivers with iommus and/or dmas DT properties where the targets of these properties do not have a driver should still be probed, eventually. But if the IOMMU or DMA drivers are present, they should be probed first, so the device drivers can make use of them.

Thanks for the logs and details.

Yeah, this is going to be a problem then. How is this handled in static kernels today? Do we just try to make sure the iommus driver probes the iommu device before the consumers? And then the consumers simply don't defer probe on failure to get iommu?

I can make this work if modules are not enabled (needs some code changes), but it's not going to work when there are modules. There's no way to tell if an iommu module won't be loaded soon. Also, device links doing this behavior only for iommu/dma is probably not a good idea. So, whatever we do will have to be common behavior. :(

Another intermediate option I was thinking was having a CONFIG_FW_DEVLINK_OFF/PERMISSIVE/ON and defaulting it to ON for ARM64 and turning it off in the defconfig for boards for which this doesn't work. That way, we can incrementally enable fw_devlink.

Quoted 8 lines

This week is a very hectic week for me. So, please bear with slow responses from me for rest of this week. Let me think about this a bit to see if I can come up with a better solution than what I have in mind.

Also, can you try deleting "iommu" and "dma" parsing in of_supplier_bindings[] in driver/of/property.c and see if it helps? Then we'd know this is the reason for things not working in your case.

Hi Geert,

I took a look at your logs. It looks like your guess is right. It's at least one of the issues.

You'll need to convert drivers/soc/renesas/rcar-sysc.c into a platform driver. You already have a platform device created for it. So just go ahead and probe it with a platform driver. See what Marek did here [1].

You probably had to implement it as an "initcall based driver" because you had to play initcall chicken to make sure the PD hardware was initialized before the consumers. With fw_devlink=on you won't have to worry about that. As an added benefit of implementing a proper platform driver, you can actually implement runtime PM now, your suspend/resume would be more robust, etc.

[1] - https://lore.kernel.org/lkml/[email protected]/

-Saravana

Quoted 14 lines
Quoted 6 lines

Thanks!

[*] Although attaching means people like myself cannot read and comment
on the log easily, without saving the attachment first. That's also the reason why patches should be submitted inline...

Yeah, I see your concern. If you want to add comments to logs when sending them, yeah, please go ahead and put it inline. Or if someone wants to add comments to what you attached, they could copy paste the relevant sections and add comments.

Thanks,
Saravana

#32 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Hi Saravana,

Quoted 122 lines · Saravana Kannan
Quoted 105 lines · Saravana Kannan
Quoted 104 lines · Geert Uytterhoeven
Quoted 73 lines · Saravana Kannan
Quoted 71 lines · Geert Uytterhoeven
Quoted 40 lines · Marc Zyngier
Quoted 28 lines · Geert Uytterhoeven
Quoted 22 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently,
only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their
status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

There is a tentative patch from Saravana here[1], which works around some issues on my RK3399 platform, and it'd be interesting to find out whether that helps on your system.

Quoted 6 lines

Thanks,

         M.

[1]
https://lore.kernel.org/r/[email protected]

Thanks for the suggestion, but given no devices probe (incl. GPIO providers), I'm afraid it won't help. [testing] Indeed.

With the debug prints in device_links_check_suppliers enabled, and some postprocessing, I get:

255 supplier e6180000.system-controller not ready
9 supplier fe990000.iommu not ready
9 supplier fe980000.iommu not ready
6 supplier febd0000.iommu not ready
6 supplier ec670000.iommu not ready
3 supplier febe0000.iommu not ready
3 supplier e7740000.iommu not ready
3 supplier e6740000.iommu not ready
3 supplier e65ee000.usb-phy not ready
3 supplier e6570000.iommu not ready
3 supplier e6054000.gpio not ready
3 supplier e6053000.gpio not ready

As everything is part of a PM Domain, the (lack of the) system controller must be the culprit. What's wrong with it? It is registered very early in the boot:

Quoted 8 lines
Quoted 1 line
[    0.142096] rcar_sysc_pd_init:442: of_genpd_add_provider_onecell() returned 0

Looks like you found the important logs. Can you please enable all these logs and send the early con logs as an attachment (so I don't need to deal with lines getting wrapped)?
1. The ones in device_links_check_suppliers()
2. The ones in device_link_add()
3. initcall_debug=1

I have attached[*] the requested log.

Quoted 2 lines

That should help us figure out what's going on. Also, what's the DT that corresponds to one of the boards that see this issue?

arch/arm64/boot/dts/renesas/r8a77951-salvator-xs.dts

Quoted 13 lines

Lastly, can you please pick up these 3 patches (some need clean up before they merge) to make sure it's not an issue being worked on from other bug reports?
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

I have a strong hunch the 2nd one will fix your issues. fw_devlink can handle cyclic dependencies now (it basically reverts to fw_devlink=permissive mode for devices in the cycle), but it needs to "see" all the dependencies to know there's a cycle. So want to make sure it "sees" the "gpios" binding used all over some of the Renesas DT files.

Quoted 7 lines
Quoted 7 lines

These patches don't help.
The 2nd one actually introduces a new failure:

     OF: /soc/i2c@e66d8000/gpio@20/pcie-sata-switch-hog: could not get

#gpio-cells for /cpus/cpu@102

Note that my issues don't seem to be GPIO-related at all.

I took a look at your logs. It looks like your guess is right. It's at least one of the issues.

You'll need to convert drivers/soc/renesas/rcar-sysc.c into a platform driver. You already have a platform device created for it. So just go ahead and probe it with a platform driver. See what Marek did here [1].

Quoted 6 lines

You probably had to implement it as an "initcall based driver" because you had to play initcall chicken to make sure the PD hardware was initialized before the consumers. With fw_devlink=on you won't have to worry about that. As an added benefit of implementing a proper platform driver, you can actually implement runtime PM now, your suspend/resume would be more robust, etc.

On R-Car H1, the system controller driver needs to be active before secondary CPU setup, hence the early_initcall(). platform_bus_init() is called after that, so this is gonna need a split initialization. Or a dummy platform driver to make devlinks think everything is fine ;-)

So basically all producer DT drivers not using a platform (or e.g. i2c) driver are now broken?
Including all clock drivers using CLK_OF_DECLARE()?

    $ git grep -L "\<[a-z0-9]*_driver\>" -- $(git grep -l
"\.compatible\>") | wc -l
    249

(includes false positives)

I doubt they'll all get fixed for v5.12, as we're already at rc4...

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#33 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Hi Saravana,

Quoted 130 lines · Geert Uytterhoeven
Quoted 122 lines · Saravana Kannan
Quoted 105 lines · Saravana Kannan
Quoted 104 lines · Geert Uytterhoeven
Quoted 73 lines · Saravana Kannan
Quoted 71 lines · Geert Uytterhoeven
Quoted 40 lines · Marc Zyngier
Quoted 28 lines · Geert Uytterhoeven
Quoted 22 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently,
only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their
status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

There is a tentative patch from Saravana here[1], which works around some issues on my RK3399 platform, and it'd be interesting to find out whether that helps on your system.

Quoted 6 lines

Thanks,

         M.

[1]
https://lore.kernel.org/r/[email protected]

Thanks for the suggestion, but given no devices probe (incl. GPIO providers), I'm afraid it won't help. [testing] Indeed.

With the debug prints in device_links_check_suppliers enabled, and some postprocessing, I get:

255 supplier e6180000.system-controller not ready
9 supplier fe990000.iommu not ready
9 supplier fe980000.iommu not ready
6 supplier febd0000.iommu not ready
6 supplier ec670000.iommu not ready
3 supplier febe0000.iommu not ready
3 supplier e7740000.iommu not ready
3 supplier e6740000.iommu not ready
3 supplier e65ee000.usb-phy not ready
3 supplier e6570000.iommu not ready
3 supplier e6054000.gpio not ready
3 supplier e6053000.gpio not ready

As everything is part of a PM Domain, the (lack of the) system controller must be the culprit. What's wrong with it? It is registered very early in the boot:

Quoted 8 lines
Quoted 1 line
[    0.142096] rcar_sysc_pd_init:442: of_genpd_add_provider_onecell() returned 0

Looks like you found the important logs. Can you please enable all these logs and send the early con logs as an attachment (so I don't need to deal with lines getting wrapped)?
1. The ones in device_links_check_suppliers()
2. The ones in device_link_add()
3. initcall_debug=1

I have attached[*] the requested log.

Quoted 2 lines

That should help us figure out what's going on. Also, what's the DT that corresponds to one of the boards that see this issue?

arch/arm64/boot/dts/renesas/r8a77951-salvator-xs.dts

Quoted 13 lines

Lastly, can you please pick up these 3 patches (some need clean up before they merge) to make sure it's not an issue being worked on from other bug reports?
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

I have a strong hunch the 2nd one will fix your issues. fw_devlink can handle cyclic dependencies now (it basically reverts to fw_devlink=permissive mode for devices in the cycle), but it needs to "see" all the dependencies to know there's a cycle. So want to make sure it "sees" the "gpios" binding used all over some of the Renesas DT files.

Quoted 7 lines
Quoted 7 lines

These patches don't help.
The 2nd one actually introduces a new failure:

     OF: /soc/i2c@e66d8000/gpio@20/pcie-sata-switch-hog: could not get

#gpio-cells for /cpus/cpu@102

Note that my issues don't seem to be GPIO-related at all.

I took a look at your logs. It looks like your guess is right. It's at least one of the issues.

You'll need to convert drivers/soc/renesas/rcar-sysc.c into a platform driver. You already have a platform device created for it. So just go ahead and probe it with a platform driver. See what Marek did here [1].

Quoted 6 lines

You probably had to implement it as an "initcall based driver" because you had to play initcall chicken to make sure the PD hardware was initialized before the consumers. With fw_devlink=on you won't have to worry about that. As an added benefit of implementing a proper platform driver, you can actually implement runtime PM now, your suspend/resume would be more robust, etc.

Quoted 5 lines

On R-Car H1, the system controller driver needs to be active before secondary CPU setup, hence the early_initcall(). platform_bus_init() is called after that, so this is gonna need a split initialization. Or a dummy platform driver to make devlinks think everything is fine ;-)

Note that adding a dummy platform driver does work.

Quoted 3 lines

So basically all producer DT drivers not using a platform (or e.g. i2c) driver are now broken?
Including all clock drivers using CLK_OF_DECLARE()?

Oh, of_link_to_phandle() ignores device nodes where OF_POPULATED is set, and of_clk_init() sets that flag. So rcar-sysc should do so, too. Patch sent.

Quoted 7 lines
    $ git grep -L "\<[a-z0-9]*_driver\>" -- $(git grep -l
"\.compatible\>") | wc -l
    249

(includes false positives)

I doubt they'll all get fixed for v5.12, as we're already at rc4...

Still more than 100 drivers to fix?

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#34 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 140 lines · Geert Uytterhoeven

Hi Saravana,

Quoted 135 lines · Geert Uytterhoeven
Quoted 122 lines · Saravana Kannan
Quoted 105 lines · Saravana Kannan
Quoted 104 lines · Geert Uytterhoeven
Quoted 73 lines · Saravana Kannan
Quoted 71 lines · Geert Uytterhoeven
Quoted 40 lines · Marc Zyngier
Quoted 28 lines · Geert Uytterhoeven
Quoted 22 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently,
only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their
status. See [1] for an example of such a case.

Quoted 3 lines

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

There is a tentative patch from Saravana here[1], which works around some issues on my RK3399 platform, and it'd be interesting to find out whether that helps on your system.

Quoted 6 lines

Thanks,

         M.

[1]
https://lore.kernel.org/r/[email protected]

Thanks for the suggestion, but given no devices probe (incl. GPIO providers), I'm afraid it won't help. [testing] Indeed.

With the debug prints in device_links_check_suppliers enabled, and some postprocessing, I get:

255 supplier e6180000.system-controller not ready
9 supplier fe990000.iommu not ready
9 supplier fe980000.iommu not ready
6 supplier febd0000.iommu not ready
6 supplier ec670000.iommu not ready
3 supplier febe0000.iommu not ready
3 supplier e7740000.iommu not ready
3 supplier e6740000.iommu not ready
3 supplier e65ee000.usb-phy not ready
3 supplier e6570000.iommu not ready
3 supplier e6054000.gpio not ready
3 supplier e6053000.gpio not ready

As everything is part of a PM Domain, the (lack of the) system controller must be the culprit. What's wrong with it? It is registered very early in the boot:

Quoted 8 lines
Quoted 1 line
[    0.142096] rcar_sysc_pd_init:442: of_genpd_add_provider_onecell() returned 0

Looks like you found the important logs. Can you please enable all these logs and send the early con logs as an attachment (so I don't need to deal with lines getting wrapped)?
1. The ones in device_links_check_suppliers()
2. The ones in device_link_add()
3. initcall_debug=1

I have attached[*] the requested log.

Quoted 2 lines

That should help us figure out what's going on. Also, what's the DT that corresponds to one of the boards that see this issue?

arch/arm64/boot/dts/renesas/r8a77951-salvator-xs.dts

Quoted 13 lines

Lastly, can you please pick up these 3 patches (some need clean up before they merge) to make sure it's not an issue being worked on from other bug reports?
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

I have a strong hunch the 2nd one will fix your issues. fw_devlink can handle cyclic dependencies now (it basically reverts to fw_devlink=permissive mode for devices in the cycle), but it needs to "see" all the dependencies to know there's a cycle. So want to make sure it "sees" the "gpios" binding used all over some of the Renesas DT files.

Quoted 7 lines
Quoted 7 lines

These patches don't help.
The 2nd one actually introduces a new failure:

     OF: /soc/i2c@e66d8000/gpio@20/pcie-sata-switch-hog: could not get

#gpio-cells for /cpus/cpu@102

Note that my issues don't seem to be GPIO-related at all.

I took a look at your logs. It looks like your guess is right. It's at least one of the issues.

You'll need to convert drivers/soc/renesas/rcar-sysc.c into a platform driver. You already have a platform device created for it. So just go ahead and probe it with a platform driver. See what Marek did here [1].

Quoted 6 lines

You probably had to implement it as an "initcall based driver" because you had to play initcall chicken to make sure the PD hardware was initialized before the consumers. With fw_devlink=on you won't have to worry about that. As an added benefit of implementing a proper platform driver, you can actually implement runtime PM now, your suspend/resume would be more robust, etc.

On R-Car H1, the system controller driver needs to be active before secondary CPU setup, hence the early_initcall(). platform_bus_init() is called after that, so this is gonna need a split initialization. Or a dummy platform driver to make devlinks think everything is fine ;-)

I was wondering if you could still probe the "not needed by CPU" power domains (if there are any) as devices. Using driver-core brings you good things :)

Quoted 15 lines

Note that adding a dummy platform driver does work.

Quoted 3 lines

So basically all producer DT drivers not using a platform (or e.g. i2c) driver are now broken?
Including all clock drivers using CLK_OF_DECLARE()?

Oh, of_link_to_phandle() ignores device nodes where OF_POPULATED is set, and of_clk_init() sets that flag. So rcar-sysc should do so, too. Patch sent.

Quoted 4 lines
    $ git grep -L "\<[a-z0-9]*_driver\>" -- $(git grep -l
"\.compatible\>") | wc -l
    249
Quoted 5 lines
Quoted 3 lines

(includes false positives)

I doubt they'll all get fixed for v5.12, as we're already at rc4...

Still more than 100 drivers to fix?

Not fully sure what the grep is trying to catch, but fw_devlink supports devices on any bus (i2c, platform, pci, etc). So that's not a problem. It'll be a problem when a struct device is never created for a real device. Or if it's created, but never probed.

I'm also looking into a bunch of other options for fallback when fw_devlink=on doesn't work. Too much to explain here -- patches are easier :)

-Saravana

#35 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Hi Saravana,

Quoted 64 lines · Saravana Kannan
Quoted 57 lines · Geert Uytterhoeven
Quoted 55 lines · Geert Uytterhoeven
Quoted 53 lines · Saravana Kannan
Quoted 34 lines · Saravana Kannan
Quoted 33 lines · Geert Uytterhoeven
Quoted 32 lines · Saravana Kannan
Quoted 30 lines · Geert Uytterhoeven
Quoted 29 lines · Marc Zyngier
Quoted 28 lines · Geert Uytterhoeven
Quoted 25 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently,
only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their
status. See [1] for an example of such a case.

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

You'll need to convert drivers/soc/renesas/rcar-sysc.c into a platform driver. You already have a platform device created for it. So just go ahead and probe it with a platform driver. See what Marek did here [1].

You probably had to implement it as an "initcall based driver" because you had to play initcall chicken to make sure the PD hardware was initialized before the consumers. With fw_devlink=on you won't have to worry about that. As an added benefit of implementing a proper platform driver, you can actually implement runtime PM now, your suspend/resume would be more robust, etc.

Quoted 5 lines
Quoted 5 lines

On R-Car H1, the system controller driver needs to be active before secondary CPU setup, hence the early_initcall(). platform_bus_init() is called after that, so this is gonna need a split initialization. Or a dummy platform driver to make devlinks think everything is fine ;-)

Quoted 3 lines

I was wondering if you could still probe the "not needed by CPU" power domains (if there are any) as devices. Using driver-core brings you good things :)

1. That would mean splitting the driver in two parts, looping over the tables twice, while everything can just be done in the first pass?

2. Which "good things" do you have in mind? Making the driver modular? Ignoring the dependency for secondary CPU setup on R-Car H1, this driver could indeed be modular on R-Car Gen2 and Gen3, as long as the boot loader would pass a ramdisk with the module to the kernel. The ramdisk could not be loaded in any other way, as all I/O devices are part of a PM Domain, and thus depend on the SYSC driver. Note that on some (non-R-Car) SoCs, the timers may be part of a PM Domain, too.

Quoted 13 lines
Quoted 13 lines
Quoted 3 lines

So basically all producer DT drivers not using a platform (or e.g. i2c) driver are now broken?
Including all clock drivers using CLK_OF_DECLARE()?

Oh, of_link_to_phandle() ignores device nodes where OF_POPULATED is set, and of_clk_init() sets that flag. So rcar-sysc should do so, too. Patch sent.

Quoted 6 lines
    $ git grep -L "\<[a-z0-9]*_driver\>" -- $(git grep -l
"\.compatible\>") | wc -l
    249

(includes false positives)

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

I doubt they'll all get fixed for v5.12, as we're already at rc4...

Still more than 100 drivers to fix?

Not fully sure what the grep is trying to catch, but fw_devlink supports devices on any bus (i2c, platform, pci, etc). So that's not a problem. It'll be a problem when a struct device is never created for a real device. Or if it's created, but never probed.

The grep tries to catch drivers using DT matching (i.e. matching ".compatible") and not using a driver model driver (i.e. not matching "*_driver").

Quoted 3 lines

I'm also looking into a bunch of other options for fallback when fw_devlink=on doesn't work. Too much to explain here -- patches are easier :)

I gave it a try on all Renesas platforms I have local access to:

- R-Car Gen2/Gen3:
Setting OF_POPULATED in the rcar-sysc driver[1] made my standard config boot again. Remaining issues:
- CONFIG_IPMMU_VMSA=n hangs: supplier fe990000.iommu not ready
- CONFIG_RCAR_DMAC=n hangs: supplier e7310000.dma-controller not ready Note that Ethernet does not use the R-Car DMAC, so DHCP works. Nevertheless, after that everything hangs, and the board does not respond to pings anymore
Both IOMMU and DMAC dependencies are optional, hence should be dropped at late boot (late_initcall?).

- SH-Mobile AG5 and R-Mobile APE6:
The rmobile-sysc driver is similar to the rcar-sysc driver, and does not use a platform device.
Still, it works, because all dependencies on the System Controller become unblocked when the rmobile-reset driver binds against the "renesas,sysc-rmobile" device. Obviously it would fail if no support for that driver is included in your kernel...

- R-Mobile A1:
Also using the rmobile-sysc driver.
However, this is a single core Cortex-A9, i.e. it does not have an ARM architectured timer (like R-Mobile APE6) or Cortex-A9 Global Timer (like SH-Mobile AG5). The timer used (TMU) is located in a PM Domain controlled by the rmobile-sysc driver, and driver initialization is postponed beyond the point where something relies on a working timer, causing a hang.

Setting OF_POPULATED (like in my fix for the rcar-sysc driver) fixes this, but prevents the rmobile-reset driver from binding against the same device node, so the reset handling will have to be incorporated into the rmobile-sysc driver (and will thus be registered very early).

- RZ/A1 and RZ/A2:
These are not affected, as the timer used (OSTM) is not a platform driver, but uses TIMER_OF_DECLARE().
Note that the RZ/A2 clock driver uses split initialization:
1. Early (timer) clocks are initialized from CLK_OF_DECLARE_DRIVER,
2. Other clocks are initialized by platform_driver_probe() from a subsys_initcall.
If the OSTM driver would be a platform_driver, it would block on the block dependency. Setting the OF_POPULATED flag in the clock driver would not work: while that flag would unblock probing of the timer driver, it would also prevent the second part of the clock driver initialization.

Now, back to the things I was supposed to work on this week ;-)

[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#36 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 75 lines · Geert Uytterhoeven

Hi Saravana,

Quoted 64 lines · Saravana Kannan
Quoted 57 lines · Geert Uytterhoeven
Quoted 55 lines · Geert Uytterhoeven
Quoted 53 lines · Saravana Kannan
Quoted 34 lines · Saravana Kannan
Quoted 33 lines · Geert Uytterhoeven
Quoted 32 lines · Saravana Kannan
Quoted 30 lines · Geert Uytterhoeven
Quoted 29 lines · Marc Zyngier
Quoted 28 lines · Geert Uytterhoeven
Quoted 25 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently,
only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their
status. See [1] for an example of such a case.

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

You'll need to convert drivers/soc/renesas/rcar-sysc.c into a platform driver. You already have a platform device created for it. So just go ahead and probe it with a platform driver. See what Marek did here [1].

You probably had to implement it as an "initcall based driver" because you had to play initcall chicken to make sure the PD hardware was initialized before the consumers. With fw_devlink=on you won't have to worry about that. As an added benefit of implementing a proper platform driver, you can actually implement runtime PM now, your suspend/resume would be more robust, etc.

Quoted 5 lines
Quoted 5 lines

On R-Car H1, the system controller driver needs to be active before secondary CPU setup, hence the early_initcall(). platform_bus_init() is called after that, so this is gonna need a split initialization. Or a dummy platform driver to make devlinks think everything is fine ;-)

Quoted 3 lines

I was wondering if you could still probe the "not needed by CPU" power domains (if there are any) as devices. Using driver-core brings you good things :)

1. That would mean splitting the driver in two parts, looping over the tables twice, while everything can just be done in the first pass?

Quoted 8 lines

2. Which "good things" do you have in mind? Making the driver modular? Ignoring the dependency for secondary CPU setup on R-Car H1, this driver could indeed be modular on R-Car Gen2 and Gen3, as long as the boot loader would pass a ramdisk with the module to the kernel. The ramdisk could not be loaded in any other way, as all I/O devices are part of a PM Domain, and thus depend on the SYSC driver. Note that on some (non-R-Car) SoCs, the timers may be part of a PM Domain, too.

"Good things" like being able to implement runtime pm, suspend/resume robustness (due to device links). There were a few more benefits I had in mind when I wrote it, but I don't remember what it was.

The double pass itself is not that big of a deal IMHO. It probably adds less than a millisecond.

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 4 lines

So basically all producer DT drivers not using a platform (or e.g. i2c) driver are now broken?
Including all clock drivers using CLK_OF_DECLARE()?

Oh, of_link_to_phandle() ignores device nodes where OF_POPULATED is set, and of_clk_init() sets that flag. So rcar-sysc should do so, too. Patch sent.

Quoted 7 lines
    $ git grep -L "\<[a-z0-9]*_driver\>" -- $(git grep -l
"\.compatible\>") | wc -l
    249

(includes false positives)

I doubt they'll all get fixed for v5.12, as we're already at rc4...

Still more than 100 drivers to fix?

Quoted 7 lines
Quoted 4 lines

Not fully sure what the grep is trying to catch, but fw_devlink supports devices on any bus (i2c, platform, pci, etc). So that's not a problem. It'll be a problem when a struct device is never created for a real device. Or if it's created, but never probed.

The grep tries to catch drivers using DT matching (i.e. matching ".compatible") and not using a driver model driver (i.e. not matching "*_driver").

Ah TIL about -L and -l. Thanks.

Quoted 5 lines
Quoted 3 lines

I'm also looking into a bunch of other options for fallback when fw_devlink=on doesn't work. Too much to explain here -- patches are easier :)

I gave it a try on all Renesas platforms I have local access to:

Thanks a lot! Really appreciate the testing and reporting.

Quoted 11 lines

- R-Car Gen2/Gen3:
Setting OF_POPULATED in the rcar-sysc driver[1] made my standard config boot again. Remaining issues:
- CONFIG_IPMMU_VMSA=n hangs: supplier fe990000.iommu not ready
- CONFIG_RCAR_DMAC=n hangs: supplier e7310000.dma-controller not ready Note that Ethernet does not use the R-Car DMAC, so DHCP works. Nevertheless, after that everything hangs, and the board does not respond to pings anymore
Both IOMMU and DMAC dependencies are optional, hence should be dropped at late boot (late_initcall?).

Yeah, I'm looking into a good/clean way of handling optional suppliers. There are a bunch of corner cases I need to consider. But in the end, I need to have it behave as closely as possible to fw_devlink=permissive.

Quoted 8 lines

- SH-Mobile AG5 and R-Mobile APE6:
The rmobile-sysc driver is similar to the rcar-sysc driver, and does not use a platform device.
Still, it works, because all dependencies on the System Controller become unblocked when the rmobile-reset driver binds against the "renesas,sysc-rmobile" device. Obviously it would fail if no support for that driver is included in your kernel...

Yeah, IMHO two real drivers (not stubs) for a single device tree node is wrong/weird at a high level. I'd think one should be a child of the other. But too late to fix that DT now.

Does it make sense for the rmobile-sysc driver to create a new platform device and have the rmobule-reset bind to that instead? And then you can bind a stub driver to the "renesas,sysc-rmobile" device? I know this can be handled by whatever solution I come up with for the IOMMU case, but that doesn't seem right for this case. We don't have to decide on this now, but that's my current view.

Quoted 9 lines

- R-Mobile A1:
Also using the rmobile-sysc driver.
However, this is a single core Cortex-A9, i.e. it does not have an ARM architectured timer (like R-Mobile APE6) or Cortex-A9 Global Timer (like SH-Mobile AG5). The timer used (TMU) is located in a PM Domain controlled by the rmobile-sysc driver, and driver initialization is postponed beyond the point where something relies on a working timer, causing a hang.

Quoted 5 lines

Setting OF_POPULATED (like in my fix for the rcar-sysc driver) fixes this, but prevents the rmobile-reset driver from binding against the same device node, so the reset handling will have to be incorporated into the rmobile-sysc driver (and will thus be registered very early).

Or you can do the "create a child device" option I suggested above.

Quoted 4 lines

- RZ/A1 and RZ/A2:
These are not affected, as the timer used (OSTM) is not a platform driver, but uses TIMER_OF_DECLARE().
Note that the RZ/A2 clock driver uses split initialization:

Quoted 8 lines

1. Early (timer) clocks are initialized from CLK_OF_DECLARE_DRIVER,
2. Other clocks are initialized by platform_driver_probe() from a subsys_initcall.
If the OSTM driver would be a platform_driver, it would block on the block dependency. Setting the OF_POPULATED flag in the clock driver would not work: while that flag would unblock probing of the timer driver, it would also prevent the second part of the clock driver initialization.

So this looks like it's all working fine, right? Yeah, I already took into account the *OF*_DECLARE macros when I wrote this and was aware of the split driver implementations. So hopefully this all works out fine.

Quoted 1 line

Now, back to the things I was supposed to work on this week ;-)

Really appreciate all this testing and feedback!

-Saravana

#37 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Hi Saravana,

Quoted 82 lines · Saravana Kannan
Quoted 72 lines · Geert Uytterhoeven
Quoted 64 lines · Saravana Kannan
Quoted 57 lines · Geert Uytterhoeven
Quoted 55 lines · Geert Uytterhoeven
Quoted 53 lines · Saravana Kannan
Quoted 34 lines · Saravana Kannan
Quoted 33 lines · Geert Uytterhoeven
Quoted 32 lines · Saravana Kannan
Quoted 30 lines · Geert Uytterhoeven
Quoted 29 lines · Marc Zyngier
Quoted 28 lines · Geert Uytterhoeven
Quoted 25 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining
reasons fw_devlink=on couldn't be set by default. Now that cyclic
dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits
(currently,
only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to
worry about functional dependencies between modules (depmod is still
needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due
to the system having one or more device drivers that "probe"/set up a
device (DT node with compatible property) without creating a struct
device for it. If we hit such cases, the device drivers need to be
fixed so that they populate struct devices and probe them like normal
device drivers so that the driver core is aware of the devices and
their
status. See [1] for an example of such a case.

[1] -
https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/
Signed-off-by: Saravana Kannan <[email protected]>

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Shimoda-san reported that next-20210111 and later fail to boot on Renesas R-Car Gen3 platforms. No output is seen, unless earlycon is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

You'll need to convert drivers/soc/renesas/rcar-sysc.c into a platform driver. You already have a platform device created for it. So just go ahead and probe it with a platform driver. See what Marek did here [1].

You probably had to implement it as an "initcall based driver" because you had to play initcall chicken to make sure the PD hardware was initialized before the consumers. With fw_devlink=on you won't have to worry about that. As an added benefit of implementing a proper platform driver, you can actually implement runtime PM now, your suspend/resume would be more robust, etc.

Quoted 5 lines
Quoted 5 lines

On R-Car H1, the system controller driver needs to be active before secondary CPU setup, hence the early_initcall(). platform_bus_init() is called after that, so this is gonna need a split initialization. Or a dummy platform driver to make devlinks think everything is fine ;-)

Quoted 3 lines

I was wondering if you could still probe the "not needed by CPU" power domains (if there are any) as devices. Using driver-core brings you good things :)

1. That would mean splitting the driver in two parts, looping over the tables twice, while everything can just be done in the first pass?

Quoted 8 lines

2. Which "good things" do you have in mind? Making the driver modular? Ignoring the dependency for secondary CPU setup on R-Car H1, this driver could indeed be modular on R-Car Gen2 and Gen3, as long as the boot loader would pass a ramdisk with the module to the kernel. The ramdisk could not be loaded in any other way, as all I/O devices are part of a PM Domain, and thus depend on the SYSC driver. Note that on some (non-R-Car) SoCs, the timers may be part of a PM Domain, too.

Quoted 3 lines

"Good things" like being able to implement runtime pm, suspend/resume robustness (due to device links). There were a few more benefits I had in mind when I wrote it, but I don't remember what it was.

While that is valid for I/O devices, the System Controller is a power provider, and thus provides Runtime PM services itself. It does not use Runtime PM itself, as it is always-on.

Note that, in theory, you can have a power provider that can be powered-down, and thus would use (need) Runtime PM, but then you need to have a second power provider that is always-on to control the first one, and the problem would just shift to the second one.

Quoted 2 lines

The double pass itself is not that big of a deal IMHO. It probably adds less than a millisecond.

Not all embedded systems run at multi-GHz speed...

Quoted 47 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 3 lines

So basically all producer DT drivers not using a platform (or e.g. i2c) driver are now broken?
Including all clock drivers using CLK_OF_DECLARE()?

Oh, of_link_to_phandle() ignores device nodes where OF_POPULATED is set, and of_clk_init() sets that flag. So rcar-sysc should do so, too. Patch sent.

Quoted 7 lines
    $ git grep -L "\<[a-z0-9]*_driver\>" -- $(git grep -l
"\.compatible\>") | wc -l
    249

(includes false positives)

I doubt they'll all get fixed for v5.12, as we're already at rc4...

Still more than 100 drivers to fix?

Quoted 7 lines
Quoted 4 lines

Not fully sure what the grep is trying to catch, but fw_devlink supports devices on any bus (i2c, platform, pci, etc). So that's not a problem. It'll be a problem when a struct device is never created for a real device. Or if it's created, but never probed.

The grep tries to catch drivers using DT matching (i.e. matching ".compatible") and not using a driver model driver (i.e. not matching "*_driver").

Ah TIL about -L and -l. Thanks.

Quoted 5 lines
Quoted 3 lines

I'm also looking into a bunch of other options for fallback when fw_devlink=on doesn't work. Too much to explain here -- patches are easier :)

I gave it a try on all Renesas platforms I have local access to:

Thanks a lot! Really appreciate the testing and reporting.

Quoted 11 lines

- R-Car Gen2/Gen3:
Setting OF_POPULATED in the rcar-sysc driver[1] made my standard config boot again. Remaining issues:
- CONFIG_IPMMU_VMSA=n hangs: supplier fe990000.iommu not ready
- CONFIG_RCAR_DMAC=n hangs: supplier e7310000.dma-controller not ready Note that Ethernet does not use the R-Car DMAC, so DHCP works. Nevertheless, after that everything hangs, and the board does not respond to pings anymore
Both IOMMU and DMAC dependencies are optional, hence should be dropped at late boot (late_initcall?).

Quoted 4 lines

Yeah, I'm looking into a good/clean way of handling optional suppliers. There are a bunch of corner cases I need to consider. But in the end, I need to have it behave as closely as possible to fw_devlink=permissive.

OK.

Quoted 12 lines
Quoted 7 lines

- SH-Mobile AG5 and R-Mobile APE6:
The rmobile-sysc driver is similar to the rcar-sysc driver, and does not use a platform device.
Still, it works, because all dependencies on the System Controller become unblocked when the rmobile-reset driver binds against the "renesas,sysc-rmobile" device. Obviously it would fail if no support for that driver is included in your kernel...

Yeah, IMHO two real drivers (not stubs) for a single device tree node is wrong/weird at a high level. I'd think one should be a child of the other. But too late to fix that DT now.

Quoted 6 lines

Does it make sense for the rmobile-sysc driver to create a new platform device and have the rmobule-reset bind to that instead? And then you can bind a stub driver to the "renesas,sysc-rmobile" device? I know this can be handled by whatever solution I come up with for the IOMMU case, but that doesn't seem right for this case. We don't have to decide on this now, but that's my current view.

I guess registering the (system) reset handler in the rmobile-sysc driver is the simplest solution. We already have clock drivers registering (device) reset support, as module clock and module reset are typically combined in the same hardware block.

Quoted 9 lines
Quoted 9 lines

- R-Mobile A1:
Also using the rmobile-sysc driver.
However, this is a single core Cortex-A9, i.e. it does not have an ARM architectured timer (like R-Mobile APE6) or Cortex-A9 Global Timer (like SH-Mobile AG5). The timer used (TMU) is located in a PM Domain controlled by the rmobile-sysc driver, and driver initialization is postponed beyond the point where something relies on a working timer, causing a hang.

Quoted 5 lines
Quoted 5 lines

Setting OF_POPULATED (like in my fix for the rcar-sysc driver) fixes this, but prevents the rmobile-reset driver from binding against the same device node, so the reset handling will have to be incorporated into the rmobile-sysc driver (and will thus be registered very early).

So the rmobile-sysc driver has to stay a DT driver.

Quoted 1 line

Or you can do the "create a child device" option I suggested above.

Registering a reset handler from the rmobile-sysc driver is fine.

Quoted 13 lines
Quoted 4 lines

- RZ/A1 and RZ/A2:
These are not affected, as the timer used (OSTM) is not a platform driver, but uses TIMER_OF_DECLARE().
Note that the RZ/A2 clock driver uses split initialization:

Quoted 8 lines

1. Early (timer) clocks are initialized from CLK_OF_DECLARE_DRIVER,
2. Other clocks are initialized by platform_driver_probe() from a subsys_initcall.
If the OSTM driver would be a platform_driver, it would block on the block dependency. Setting the OF_POPULATED flag in the clock driver would not work: while that flag would unblock probing of the timer driver, it would also prevent the second part of the clock driver initialization.

Quoted 4 lines

So this looks like it's all working fine, right? Yeah, I already took into account the *OF*_DECLARE macros when I wrote this and was aware of the split driver implementations. So hopefully this all works out fine.

Some of it is working by accident.

I expect there are systems where the timer driver has been converted from TIMER_OF_DECLARE() to a platform driver (which people are recommending, as it is needed for Runtime PM support etc.), and that will break. It's hard to predict.

I tested on all Renesas boards I had, as I expected to discover breakage. But what exactly broke, and why, was sometimes a bit of a surprise to me ;-)

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#38 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

In reply to Saravana Kannan

Hi Saravana,

Quoted 11 lines · Saravana Kannan
Quoted 5 lines · Geert Uytterhoeven

BTW, you are aware IOMMUs and DMA controllers are optional? I.e. device drivers with iommus and/or dmas DT properties where the targets of these properties do not have a driver should still be probed, eventually. But if the IOMMU or DMA drivers are present, they should be probed first, so the device drivers can make use of them.

Yeah, this is going to be a problem then. How is this handled in static kernels today? Do we just try to make sure the iommus driver probes the iommu device before the consumers? And then the consumers simply don't defer probe on failure to get iommu?

Iommus are handled by the iommu framework, not by the driver. So the framework decides if/when it's OK to probe a device tied to an iommu. Hence the consumers' drivers don't return -EPROBE_DEFER, the framework takes care of that, before drivers' probe() functions are called.

DMA is handled by consumer drivers, and driver-specific. Many consumer drivers consider DMA optional, and fall back to PIO if getting the DMA channel failed. Some drivers retry getting the DMA channel when the device is used, and thus may start using DMA when the DMAC driver appears and probes.

Quoted 5 lines

I can make this work if modules are not enabled (needs some code changes), but it's not going to work when there are modules. There's no way to tell if an iommu module won't be loaded soon. Also, device links doing this behavior only for iommu/dma is probably not a good idea. So, whatever we do will have to be common behavior. :(

The iommu driver definitely needs to be built-in. Modular DMAC drivers currently work with consumer drivers that either consider DMA mandatory, or retry obtaining DMA channels.

Quoted 3 lines

Also, can you try deleting "iommu" and "dma" parsing in of_supplier_bindings[] in driver/of/property.c and see if it helps? Then we'd know this is the reason for things not working in your case.

It also fails on another system without "iommus" properties:

    182 supplier e6180000.system-controller not ready
     18 supplier e6055400.gpio not ready
     15 supplier e6055800.gpio not ready
     15 supplier e6052000.gpio not ready
      6 supplier e6055000.gpio not ready

The system controller is the culprit, and is a dependency for all devices due to power-domains.

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#39 lore raw

PATCH 1/1 driver: core: Make fw_devlink=on more forgiving

In reply to Saravana Kannan

This patch is for test purposes only and pretty experimental. Code might not be optimized, clean, formatted properly, etc.

Please review it only for functional bugs like locking bugs, wrong logic, etc.

It's basically trying to figure out which devices will never probe and ignore them. Might not always work.

Marek, Geert, Marc,

Can you please try this patch INSTEAD of the other workarounds we found?

Jon, Michael,

I'm explicitly not including you in the "To" because this patch won't work for your issues.

Cc Marek Szyprowski <[email protected]>
Cc Geert Uytterhoeven <[email protected]>
Cc Marc Zyngier <[email protected]>
---
 drivers/base/base.h |   3 ++
 drivers/base/core.c | 117 +++++++++++++++++++++++++++++++++++++++++++-
 drivers/base/dd.c   |  24 +++++++++
 3 files changed, 142 insertions(+), 2 deletions(-)
drivers/base/base.h +3 -0
index f5600a83124f..8d5fd95fa147 100644
@@ -106,6 +106,9 @@ struct device_private {
 #define to_device_private_class(obj)	\
 	container_of(obj, struct device_private, knode_class)
 
+bool fw_devlink_is_permissive(void);
+bool fw_devlink_unblock_probe(struct device *dev);
+
 /* initialisation functions */
 extern int devices_init(void);
 extern int buses_init(void);
drivers/base/core.c +115 -2

135 lines not on this page · raw message

drivers/base/dd.c +24 -0
index 2f32f38a11ed..d4ccd2a2b6a4 100644
@@ -301,6 +301,25 @@ static void deferred_probe_timeout_work_func(struct work_struct *work)
 }
 static DECLARE_DELAYED_WORK(deferred_probe_timeout_work, deferred_probe_timeout_work_func);
 
+static bool deferred_probe_fw_devlink_unblock(void)
+{
+	struct device *dev;
+	struct device_private *private;
+	bool unblocked = false;
+
+	if (!fw_devlink_get_flags() || fw_devlink_is_permissive())
+		return false;
+
+	mutex_lock(&deferred_probe_mutex);
+	list_for_each_entry(private, &deferred_probe_pending_list, deferred_probe) {
+		dev = private->device;
+		unblocked |= fw_devlink_unblock_probe(dev);
+	}
+	mutex_unlock(&deferred_probe_mutex);
+
+	return unblocked;
+}
+
 /**
  * deferred_probe_initcall() - Enable probing of deferred devices
  *
@@ -317,6 +336,11 @@ static int deferred_probe_initcall(void)
 	driver_deferred_probe_trigger();
 	/* Sort as many dependencies as possible before exiting initcalls */
 	flush_work(&deferred_probe_work);
+
+	while (deferred_probe_fw_devlink_unblock()) {
+		driver_deferred_probe_trigger();
+		flush_work(&deferred_probe_work);
+	}
 	initcalls_done = true;
 
 	/*
Signature · 1 line
2.30.0.296.g2bfb1c46d8-goog
#40 lore raw

Re: [TEST PATCH v1] driver: core: Make fw_devlink=on more forgiving

Quoted 7 lines · Saravana Kannan

This patch is for test purposes only and pretty experimental. Code might not be optimized, clean, formatted properly, etc.

Please review it only for functional bugs like locking bugs, wrong logic, etc.

Quoted 6 lines

It's basically trying to figure out which devices will never probe and ignore them. Might not always work.

Marek, Geert, Marc,

Can you please try this patch INSTEAD of the other workarounds we found?

Oh and can you please also try with the CONFIG_MODULES enabled vs disabled? Or have it disabled but fix the patch so the condition always evaluates to true.

Thanks,
Saravana

Quoted 217 lines

Jon, Michael,

I'm explicitly not including you in the "To" because this patch won't work for your issues.

Cc Marek Szyprowski <[email protected]>
Cc Geert Uytterhoeven <[email protected]>
Cc Marc Zyngier <[email protected]>
---
 drivers/base/base.h |   3 ++
 drivers/base/core.c | 117 +++++++++++++++++++++++++++++++++++++++++++-
 drivers/base/dd.c   |  24 +++++++++
 3 files changed, 142 insertions(+), 2 deletions(-)
drivers/base/base.h +3 -0
index f5600a83124f..8d5fd95fa147 100644
@@ -106,6 +106,9 @@ struct device_private {
 #define to_device_private_class(obj)   \
        container_of(obj, struct device_private, knode_class)
 
+bool fw_devlink_is_permissive(void);
+bool fw_devlink_unblock_probe(struct device *dev);
+
 /* initialisation functions */
 extern int devices_init(void);
 extern int buses_init(void);
drivers/base/core.c +115 -2
index e61e62b624ce..8528704bbb40 100644
@@ -49,7 +49,6 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);
 static LIST_HEAD(deferred_sync);
 static unsigned int defer_sync_state_count = 1;
 static DEFINE_MUTEX(fwnode_link_lock);
-static bool fw_devlink_is_permissive(void);
 
 /**
  * fwnode_link_add - Create a link between two fwnode_handles.
@@ -1481,7 +1480,7 @@ u32 fw_devlink_get_flags(void)
        return fw_devlink_flags;
 }
 
-static bool fw_devlink_is_permissive(void)
+bool fw_devlink_is_permissive(void)
 {
        return fw_devlink_flags == FW_DEVLINK_FLAGS_PERMISSIVE;
 }
@@ -1552,6 +1551,120 @@ static int fw_devlink_relax_cycle(struct device *con, void *sup)
        return ret;
 }
 
+static int __device_links_suppliers_available(struct device *dev)
+{
+       struct device_link *link;
+       int ret = 0;
+
+       if (dev->fwnode && !list_empty(&dev->fwnode->suppliers) &&
+           !fw_devlink_is_permissive()) {
+               return -EPROBE_DEFER;
+       }
+
+       list_for_each_entry(link, &dev->links.suppliers, c_node) {
+               if (!(link->flags & DL_FLAG_MANAGED))
+                       continue;
+
+               if (link->status != DL_STATE_AVAILABLE &&
+                   !(link->flags & DL_FLAG_SYNC_STATE_ONLY)) {
+                       ret = -EPROBE_DEFER;
+                       break;
+               }
+       }
+
+       return ret;
+}
+
+bool fw_devlink_unblock_probe(struct device *dev)
+{
+       struct fwnode_link *link, *tmp;
+       struct device_link *dev_link, *dev_ln;
+       struct fwnode_handle *fwnode = dev->fwnode;
+       bool unblocked = false;
+
+       if (!fw_devlink_get_flags() || fw_devlink_is_permissive())
+               return false;
+
+       if (!fwnode)
+               return false;
+
+       mutex_lock(&fwnode_link_lock);
+
+       /* Delete questionable fwnode links */
+       list_for_each_entry_safe(link, tmp, &fwnode->suppliers, c_hook) {
+               struct device *par_dev;
+               struct fwnode_handle *par;
+               bool bound;
+
+               /*
+                * Walk up fwnode tree of supplier till we find a parent device
+                * that has been added or a parent fwnode that has fwnode links
+                * (this is a firmware node that is expected to be added as a
+                * device in the future).
+                */
+               par = fwnode_get_parent(link->supplier);
+               while (par && list_empty(&par->suppliers) && !par->dev)
+                       par = fwnode_get_next_parent(par);
+
+               /* Supplier is waiting on parent device to be added. */
+               if (par && !par->dev) {
+                       fwnode_handle_put(par);
+                       continue;
+               }
+
+               if (par && par->dev) {
+                       par_dev = get_dev_from_fwnode(fwnode);
+                       device_lock(par_dev);
+                       bound = device_is_bound(par_dev);
+                       device_unlock(par_dev);
+                       put_device(par_dev);
+
+                       /* Supplier is waiting on parent device to be bound. */
+                       if (!bound)
+                               continue;
+               }
+
+               /*
+                * Supplier has no parent or the immediate parent device has
+                * been bound to a device. It should have been added by now.
+                * So, this link is spurious. Delete it.
+                */
+               dev_info(dev, "Deleting fwnode link to %pfwP\n",
+                        link->supplier);
+               list_del(&link->s_hook);
+               list_del(&link->c_hook);
+               kfree(link);
+               unblocked = true;
+       }
+
+       if (IS_ENABLED(CONFIG_MODULES))
+               goto out;
+
+       device_links_write_lock();
+
+       list_for_each_entry_safe(dev_link, dev_ln, &dev->links.suppliers,
+                                c_node) {
+               if (!(dev_link->flags & DL_FLAG_INFERRED) ||
+                   dev_link->flags & DL_FLAG_SYNC_STATE_ONLY ||
+                   dev_link->status != DL_STATE_DORMANT)
+                       continue;
+
+               /* This supplier should have probed by now. */
+               if (!__device_links_suppliers_available(dev_link->supplier)) {
+                       dev_info(dev, "Deleting dev link to %s\n",
+                                dev_name(dev_link->supplier));
+                       device_link_drop_managed(dev_link);
+                       unblocked = true;
+               }
+       }
+
+       device_links_write_unlock();
+
+out:
+       mutex_unlock(&fwnode_link_lock);
+       return unblocked;
+}
+
 /**
  * fw_devlink_create_devlink - Create a device link from a consumer to fwnode
  * @con - Consumer device for the device link
drivers/base/dd.c +24 -1
index 2f32f38a11ed..d4ccd2a2b6a4 100644
@@ -301,6 +301,25 @@ static void deferred_probe_timeout_work_func(struct work_struct *work)
 }
 static DECLARE_DELAYED_WORK(deferred_probe_timeout_work, deferred_probe_timeout_work_func);
 
+static bool deferred_probe_fw_devlink_unblock(void)
+{
+       struct device *dev;
+       struct device_private *private;
+       bool unblocked = false;
+
+       if (!fw_devlink_get_flags() || fw_devlink_is_permissive())
+               return false;
+
+       mutex_lock(&deferred_probe_mutex);
+       list_for_each_entry(private, &deferred_probe_pending_list, deferred_probe) {
+               dev = private->device;
+               unblocked |= fw_devlink_unblock_probe(dev);
+       }
+       mutex_unlock(&deferred_probe_mutex);
+
+       return unblocked;
+}
+
 /**
  * deferred_probe_initcall() - Enable probing of deferred devices
  *
@@ -317,6 +336,11 @@ static int deferred_probe_initcall(void)
        driver_deferred_probe_trigger();
        /* Sort as many dependencies as possible before exiting initcalls */
        flush_work(&deferred_probe_work);
+
+       while (deferred_probe_fw_devlink_unblock()) {
+               driver_deferred_probe_trigger();
+               flush_work(&deferred_probe_work);
+       }
        initcalls_done = true;
 
        /*
--

2.30.0.296.g2bfb1c46d8-goog

#41 lore raw

Re: [TEST PATCH v1] driver: core: Make fw_devlink=on more forgiving

Hi Saravana,

Quoted 7 lines · Saravana Kannan
Quoted 6 lines · Saravana Kannan

This patch is for test purposes only and pretty experimental. Code might not be optimized, clean, formatted properly, etc.

Please review it only for functional bugs like locking bugs, wrong logic, etc.

Quoted 6 lines
Quoted 6 lines

It's basically trying to figure out which devices will never probe and ignore them. Might not always work.

Marek, Geert, Marc,

Can you please try this patch INSTEAD of the other workarounds we found?

Thanks for your patch!

Quoted 3 lines

Oh and can you please also try with the CONFIG_MODULES enabled vs disabled? Or have it disabled but fix the patch so the condition always evaluates to true.

With CONFIG_MODULES=y, it fails in the same way as before (no "Deleting {fwnode,dev} link" messages seen), with a new lockdep warning:

+======================================================
+WARNING: possible circular locking dependency detected +5.11.0-rc2-salvator-x-00009-gdf1dd3208a90 #941 Not tainted
+------------------------------------------------------
+swapper/0/1 is trying to acquire lock:
+ffffffc0110da488 (fwnode_link_lock){+.+.}-{4:4}, at: fw_devlink_unblock_probe+0 x50/0x158
+
+but task is already holding lock:
+ffffffc0110da988 (deferred_probe_mutex){+.+.}-{4:4}, at: deferred_probe_initcal l+0xe4/0x12c
+
+which lock already depends on the new lock.
+
+
+the existing dependency chain (in reverse order) is:
+

+-> #2 (deferred_probe_mutex){+.+.}-{4:4}:
+       lock_acquire+0x344/0x390
+       __mutex_lock+0xc0/0x37c
+       mutex_lock_nested+0x34/0x48
+       driver_deferred_probe_add+0x2c/0x88
+       device_links_driver_bound+0x11c/0x1ac
+       driver_bound+0x64/0xac
+       really_probe+0x304/0x338
+       driver_probe_device+0x98/0xa8
+       device_driver_attach+0x40/0x68
+       __driver_attach+0xa8/0xac
+       bus_for_each_dev+0x6c/0xb8
+       driver_attach+0x20/0x28
+       bus_add_driver+0x16c/0x1b0
+       driver_register+0xac/0xe4
+       __platform_driver_probe+0x88/0xe0
+       cpg_mssr_init+0x20/0x28
+       do_one_initcall+0xf0/0x280
+       kernel_init_freeable+0x1e0/0x1e4
+       kernel_init+0x10/0x108
+       ret_from_fork+0x10/0x18
+
+-> #1 (device_links_lock){+.+.}-{4:4}:
+       lock_acquire+0x344/0x390
+       __mutex_lock+0xc0/0x37c
+       mutex_lock_nested+0x34/0x48
+       device_links_write_lock+0x18/0x20
+       device_link_add+0xfc/0x3e4
+       fw_devlink_create_devlink+0x40/0xec
+       device_add+0x640/0x6ac
+       of_device_add+0x38/0x40
+       of_platform_device_create_pdata+0xb0/0xcc
+       of_platform_bus_create+0x2b8/0x364
+       of_platform_bus_create+0x300/0x364
+       of_platform_populate+0x7c/0xd8
+       of_platform_default_populate+0x20/0x28
+       of_platform_default_populate_init+0x80/0xb8
+       do_one_initcall+0xf0/0x280
+       kernel_init_freeable+0x1e0/0x1e4
+       kernel_init+0x10/0x108
+       ret_from_fork+0x10/0x18
+
+-> #0 (fwnode_link_lock){+.+.}-{4:4}:
+       check_noncircular+0x74/0xa4
+       __lock_acquire+0xdd0/0x10a8
+       lock_acquire+0x344/0x390
+       __mutex_lock+0xc0/0x37c
+       mutex_lock_nested+0x34/0x48
+       fw_devlink_unblock_probe+0x50/0x158
+       deferred_probe_initcall+0x11c/0x12c
+       do_one_initcall+0xf0/0x280
+       kernel_init_freeable+0x1e0/0x1e4
+       kernel_init+0x10/0x108
+       ret_from_fork+0x10/0x18

+
+other info that might help us debug this:
+
+Chain exists of:
+ fwnode_link_lock --> device_links_lock --> deferred_probe_mutex +
+ Possible unsafe locking scenario:
+

+       CPU0                    CPU1
+       ----                    ----

+ lock(deferred_probe_mutex);
+ lock(device_links_lock);
+ lock(deferred_probe_mutex);
+ lock(fwnode_link_lock);
+
+ *** DEADLOCK ***
+
+1 lock held by swapper/0/1:
+ #0: ffffffc0110da988 (deferred_probe_mutex){+.+.}-{4:4}, at:
deferred_probe_initcall+0xe4/0x12c
+
+stack backtrace:
+CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.11.0-rc2-salvator-x-00009-gdf1dd3208a90 #941
+Hardware name: Renesas Salvator-X 2nd version board based on r8a77951 (DT) +Call trace:
+ dump_backtrace+0x0/0x188
+ show_stack+0x14/0x28
+ dump_stack+0xf0/0x140
+ print_circular_bug.isra.0+0x1b0/0x1e8
+ check_noncircular+0x74/0xa4
+ __lock_acquire+0xdd0/0x10a8
+ lock_acquire+0x344/0x390
+ __mutex_lock+0xc0/0x37c
+ mutex_lock_nested+0x34/0x48
+ fw_devlink_unblock_probe+0x50/0x158
+ deferred_probe_initcall+0x11c/0x12c
+ do_one_initcall+0xf0/0x280
+ kernel_init_freeable+0x1e0/0x1e4
+ kernel_init+0x10/0x108
+ ret_from_fork+0x10/0x18

With CONFIG_MODULES disabled, it deletes one link, and hangs:

+platform e61c0000.interrupt-controller: Deleting dev link to e6180000.system-controller
+INFO: task swapper/0:1 blocked for more than 120 seconds.
+ Not tainted 5.11.0-rc2-salvator-x-00009-gdf1dd3208a90-dirty #944
+"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
+task:swapper/0 state:D stack: 0 pid: 1 ppid: 0 flags:0x00000008
+Call trace:
+ __switch_to+0xa8/0x10c
+ __schedule+0x54c/0x728
+ schedule+0x7c/0xc0
+ schedule_preempt_disabled+0x10/0x1c
+ __mutex_lock+0x1e8/0x37c
+ mutex_lock_nested+0x34/0x48
+ driver_deferred_probe_del+0x28/0x8c
+ device_del+0x198/0x30c
+ device_unregister+0x14/0x28
+ __device_link_del+0x4c/0x5c
+ device_link_drop_managed+0x44/0x50
+ fw_devlink_unblock_probe+0x1e8/0x230
+ deferred_probe_initcall+0x11c/0x12c
+ do_one_initcall+0xf0/0x240
+ kernel_init_freeable+0x1e0/0x1e4
+ kernel_init+0x10/0x108
+ ret_from_fork+0x10/0x18
+INFO: lockdep is turned off.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#42 lore raw

Re: [TEST PATCH v1] driver: core: Make fw_devlink=on more forgiving

Quoted 168 lines · Geert Uytterhoeven

Hi Saravana,

Quoted 7 lines · Saravana Kannan
Quoted 6 lines · Saravana Kannan

This patch is for test purposes only and pretty experimental. Code might not be optimized, clean, formatted properly, etc.

Please review it only for functional bugs like locking bugs, wrong logic, etc.

Quoted 6 lines
Quoted 6 lines

It's basically trying to figure out which devices will never probe and ignore them. Might not always work.

Marek, Geert, Marc,

Can you please try this patch INSTEAD of the other workarounds we found?

Thanks for your patch!

Quoted 3 lines

Oh and can you please also try with the CONFIG_MODULES enabled vs disabled? Or have it disabled but fix the patch so the condition always evaluates to true.

With CONFIG_MODULES=y, it fails in the same way as before (no "Deleting {fwnode,dev} link" messages seen), with a new lockdep warning:

+======================================================

(quoted diff) +5 -0
+WARNING: possible circular locking dependency detected
+5.11.0-rc2-salvator-x-00009-gdf1dd3208a90 #941 Not tainted
+------------------------------------------------------
+swapper/0/1 is trying to acquire lock:
+ffffffc0110da488 (fwnode_link_lock){+.+.}-{4:4}, at: fw_devlink_unblock_probe+0

x50/0x158

(quoted diff) +3 -0
+
+but task is already holding lock:
+ffffffc0110da988 (deferred_probe_mutex){+.+.}-{4:4}, at: deferred_probe_initcal

l+0xe4/0x12c

(quoted diff) +79 -0
+
+which lock already depends on the new lock.
+
+
+the existing dependency chain (in reverse order) is:
+
+-> #2 (deferred_probe_mutex){+.+.}-{4:4}:
+       lock_acquire+0x344/0x390
+       __mutex_lock+0xc0/0x37c
+       mutex_lock_nested+0x34/0x48
+       driver_deferred_probe_add+0x2c/0x88
+       device_links_driver_bound+0x11c/0x1ac
+       driver_bound+0x64/0xac
+       really_probe+0x304/0x338
+       driver_probe_device+0x98/0xa8
+       device_driver_attach+0x40/0x68
+       __driver_attach+0xa8/0xac
+       bus_for_each_dev+0x6c/0xb8
+       driver_attach+0x20/0x28
+       bus_add_driver+0x16c/0x1b0
+       driver_register+0xac/0xe4
+       __platform_driver_probe+0x88/0xe0
+       cpg_mssr_init+0x20/0x28
+       do_one_initcall+0xf0/0x280
+       kernel_init_freeable+0x1e0/0x1e4
+       kernel_init+0x10/0x108
+       ret_from_fork+0x10/0x18
+
+-> #1 (device_links_lock){+.+.}-{4:4}:
+       lock_acquire+0x344/0x390
+       __mutex_lock+0xc0/0x37c
+       mutex_lock_nested+0x34/0x48
+       device_links_write_lock+0x18/0x20
+       device_link_add+0xfc/0x3e4
+       fw_devlink_create_devlink+0x40/0xec
+       device_add+0x640/0x6ac
+       of_device_add+0x38/0x40
+       of_platform_device_create_pdata+0xb0/0xcc
+       of_platform_bus_create+0x2b8/0x364
+       of_platform_bus_create+0x300/0x364
+       of_platform_populate+0x7c/0xd8
+       of_platform_default_populate+0x20/0x28
+       of_platform_default_populate_init+0x80/0xb8
+       do_one_initcall+0xf0/0x280
+       kernel_init_freeable+0x1e0/0x1e4
+       kernel_init+0x10/0x108
+       ret_from_fork+0x10/0x18
+
+-> #0 (fwnode_link_lock){+.+.}-{4:4}:
+       check_noncircular+0x74/0xa4
+       __lock_acquire+0xdd0/0x10a8
+       lock_acquire+0x344/0x390
+       __mutex_lock+0xc0/0x37c
+       mutex_lock_nested+0x34/0x48
+       fw_devlink_unblock_probe+0x50/0x158
+       deferred_probe_initcall+0x11c/0x12c
+       do_one_initcall+0xf0/0x280
+       kernel_init_freeable+0x1e0/0x1e4
+       kernel_init+0x10/0x108
+       ret_from_fork+0x10/0x18
+
+other info that might help us debug this:
+
+Chain exists of:
+  fwnode_link_lock --> device_links_lock --> deferred_probe_mutex
+
+ Possible unsafe locking scenario:
+
+       CPU0                    CPU1
+       ----                    ----
+  lock(deferred_probe_mutex);
+                               lock(device_links_lock);
+                               lock(deferred_probe_mutex);
+  lock(fwnode_link_lock);
+
+ *** DEADLOCK ***
+
+1 lock held by swapper/0/1:
+ #0: ffffffc0110da988 (deferred_probe_mutex){+.+.}-{4:4}, at:

deferred_probe_initcall+0xe4/0x12c

(quoted diff) +3 -0
+
+stack backtrace:
+CPU: 2 PID: 1 Comm: swapper/0 Not tainted

5.11.0-rc2-salvator-x-00009-gdf1dd3208a90 #941

(quoted diff) +17 -0
+Hardware name: Renesas Salvator-X 2nd version board based on r8a77951 (DT)
+Call trace:
+ dump_backtrace+0x0/0x188
+ show_stack+0x14/0x28
+ dump_stack+0xf0/0x140
+ print_circular_bug.isra.0+0x1b0/0x1e8
+ check_noncircular+0x74/0xa4
+ __lock_acquire+0xdd0/0x10a8
+ lock_acquire+0x344/0x390
+ __mutex_lock+0xc0/0x37c
+ mutex_lock_nested+0x34/0x48
+ fw_devlink_unblock_probe+0x50/0x158
+ deferred_probe_initcall+0x11c/0x12c
+ do_one_initcall+0xf0/0x280
+ kernel_init_freeable+0x1e0/0x1e4
+ kernel_init+0x10/0x108
+ ret_from_fork+0x10/0x18

With CONFIG_MODULES disabled, it deletes one link, and hangs:

+platform e61c0000.interrupt-controller: Deleting dev link to e6180000.system-controller

(quoted diff) +23 -0
+INFO: task swapper/0:1 blocked for more than 120 seconds.
+      Not tainted 5.11.0-rc2-salvator-x-00009-gdf1dd3208a90-dirty #944
+"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
+task:swapper/0       state:D stack:    0 pid:    1 ppid:     0 flags:0x00000008
+Call trace:
+ __switch_to+0xa8/0x10c
+ __schedule+0x54c/0x728
+ schedule+0x7c/0xc0
+ schedule_preempt_disabled+0x10/0x1c
+ __mutex_lock+0x1e8/0x37c
+ mutex_lock_nested+0x34/0x48
+ driver_deferred_probe_del+0x28/0x8c
+ device_del+0x198/0x30c
+ device_unregister+0x14/0x28
+ __device_link_del+0x4c/0x5c
+ device_link_drop_managed+0x44/0x50
+ fw_devlink_unblock_probe+0x1e8/0x230
+ deferred_probe_initcall+0x11c/0x12c
+ do_one_initcall+0xf0/0x240
+ kernel_init_freeable+0x1e0/0x1e4
+ kernel_init+0x10/0x108
+ ret_from_fork+0x10/0x18
+INFO: lockdep is turned off.
Quoted 3 lines

Gr{oetje,eeting}s,

Geert

Thanks for the tests Marek and Geert! I'll probably scrap this and redo it.

-Saravana

#43 lore raw

Re: [TEST PATCH v1] driver: core: Make fw_devlink=on more forgiving

In reply to Saravana Kannan

Hi Saravana,

Quoted 6 lines · Saravana Kannan

This patch is for test purposes only and pretty experimental. Code might not be optimized, clean, formatted properly, etc.

Please review it only for functional bugs like locking bugs, wrong logic, etc.

Quoted 6 lines

It's basically trying to figure out which devices will never probe and ignore them. Might not always work.

Marek, Geert, Marc,

Can you please try this patch INSTEAD of the other workarounds we found?

I've checked the latest linux-next with this patch and commit c09a3e6c97f0 ("soc: samsung: pm_domains: Convert to regular platform driver") reverted. Sadly it doesn't help. All devices that belongs to the Exynos power domains are not probed at all ("supplier 10023cXX.power-domain not ready").

Quoted 9 lines

Jon, Michael,

I'm explicitly not including you in the "To" because this patch won't work for your issues.

Cc Marek Szyprowski <[email protected]>
Cc Geert Uytterhoeven <[email protected]>
Cc Marc Zyngier <[email protected]>
Quoted 7 lines
---
  drivers/base/base.h |   3 ++
  drivers/base/core.c | 117 +++++++++++++++++++++++++++++++++++++++++++-
  drivers/base/dd.c   |  24 +++++++++
  3 files changed, 142 insertions(+), 2 deletions(-)
Quoted 1 line

[...]

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#44 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

In reply to Saravana Kannan

Hi, Saravana,

Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

next-20210125 fails to boot on at91 sama5d2 platforms. No output is seen, unless earlyprintk is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

I've attached a log that I'm seeing on a sama5d2_xplained (sama5_defconfig and arch/arm/boot/dts/at91-sama5d2_xplained.dts). I enabled the following logs:
1. The ones in device_links_check_suppliers()
2. The ones in device_link_add()
3. initcall_debug=1

There seem to be some probe fails due to the pmc supplier not being ready:
calling at_xdmac_init+0x0/0x18 @ 1
platform f0010000.dma-controller: probe deferral - supplier f0014000.pmc not ready platform f0004000.dma-controller: probe deferral - supplier f0014000.pmc not ready initcall at_xdmac_init+0x0/0x18 returned -19 after 19531 usecs

calling udc_driver_init+0x0/0x18 @ 1
platform 300000.gadget: probe deferral - supplier f0014000.pmc not ready initcall udc_driver_init+0x0/0x18 returned -19 after 7524 usecs

There are others too. I'm checking them.

Cheers,
ta

Starting kernel ...

Booting Linux on physical CPU 0x0
Linux version 5.11.0-rc4-next-20210125+ (atudor@atudor-ThinkPad-T470p) (arm-linux-gnueabi-gcc (Linaro GCC 7.5-2019.12) 7.5.0, GNU ld (Linaro_Binutils-2019.12) 2.28.2.20170706) #57 Mon Jan 25 18:36:05 EET 2021
CPU: ARMv7 Processor [410fc051] revision 1 (ARMv7), cr=10c53c7d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
OF: fdt: Machine model: Atmel SAMA5D2 Xplained
printk: debug: ignoring loglevel setting.
printk: bootconsole [earlycon0] enabled
Memory policy: Data cache writeback
Zone ranges:
Normal [mem 0x0000000020000000-0x000000003fffffff]
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000020000000-0x000000003fffffff]
Initmem setup node 0 [mem 0x0000000020000000-0x000000003fffffff]
On node 0 totalpages: 131072
Normal zone: 1024 pages used for memmap
Normal zone: 0 pages reserved
Normal zone: 131072 pages, LIFO batch:31
CPU: All CPU(s) started in SVC mode.
pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
pcpu-alloc: [0] 0
Built 1 zonelists, mobility grouping on. Total pages: 130048
Kernel command line: console=ttyS0,115200 root=/dev/nfs rootfstype=nfs rw rootwait ignore_loglevel earlyprintk initcall_debug nfsroot=192.168.0.120:/nfsroot/2020.04/sama5d2-xplained,tcp,v3 ip=192.168.0.125:192.168.0.120::
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 506940K/524288K available (8192K kernel code, 281K rwdata, 1896K rodata, 1024K init, 113K bss, 17348K reserved, 0K cma-reserved)
NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
L2C-310 ID prefetch enabled, offset 2 lines
L2C-310 dynamic clock gating enabled, standby mode enabled
L2C-310 cache controller enabled, 8 ways, 128 kB
L2C-310: CACHE_ID 0x410000c9, AUX_CTRL 0x36020000
random: get_random_bytes called from start_kernel+0x33c/0x4c0 with crng_init=0
clocksource: timer@f800c000: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 184217874325 ns
sched_clock: 32 bits at 10MHz, resolution 96ns, wraps every 206986376143ns
Switching to timer-based delay loop, resolution 96ns
Console: colour dummy device 80x30
Calibrating delay loop (skipped), value calculated using timer frequency.. 20.75 BogoMIPS (lpj=103750)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
CPU: Testing write buffer coherency: ok
calling cpu_suspend_alloc_sp+0x0/0xb4 @ 1
initcall cpu_suspend_alloc_sp+0x0/0xb4 returned 0 after 0 usecs
calling init_static_idmap+0x0/0x10c @ 1
Setting up static identity map for 0x20100000 - 0x20100060
initcall init_static_idmap+0x0/0x10c returned 0 after 9765 usecs
calling spawn_ksoftirqd+0x0/0x48 @ 1
initcall spawn_ksoftirqd+0x0/0x48 returned 0 after 0 usecs
calling initialize_ptr_random+0x0/0x54 @ 1
initcall initialize_ptr_random+0x0/0x54 returned 0 after 0 usecs
devtmpfs: initialized
calling ipc_ns_init+0x0/0x3c @ 1
initcall ipc_ns_init+0x0/0x3c returned 0 after 0 usecs
calling init_mmap_min_addr+0x0/0x20 @ 1
initcall init_mmap_min_addr+0x0/0x20 returned 0 after 0 usecs
calling net_ns_init+0x0/0x294 @ 1
initcall net_ns_init+0x0/0x294 returned 0 after 0 usecs
calling inet_frag_wq_init+0x0/0x4c @ 1
initcall inet_frag_wq_init+0x0/0x4c returned 0 after 0 usecs
calling vfp_init+0x0/0x1d4 @ 1
VFP support v0.3: implementor 41 architecture 2 part 30 variant 5 rev 1
initcall vfp_init+0x0/0x1d4 returned 0 after 9765 usecs
calling ptrace_break_init+0x0/0x2c @ 1
initcall ptrace_break_init+0x0/0x2c returned 0 after 0 usecs
calling v6_userpage_init+0x0/0x8 @ 1
initcall v6_userpage_init+0x0/0x8 returned 0 after 0 usecs
calling wq_sysfs_init+0x0/0x30 @ 1
initcall wq_sysfs_init+0x0/0x30 returned 0 after 0 usecs
calling ksysfs_init+0x0/0xa4 @ 1
initcall ksysfs_init+0x0/0xa4 returned 0 after 0 usecs
calling pm_init+0x0/0x78 @ 1
initcall pm_init+0x0/0x78 returned 0 after 0 usecs
calling rcu_set_runtime_mode+0x0/0x18 @ 1
initcall rcu_set_runtime_mode+0x0/0x18 returned 0 after 0 usecs
calling dma_init_reserved_memory+0x0/0x5c @ 1
initcall dma_init_reserved_memory+0x0/0x5c returned -12 after 0 usecs
calling init_jiffies_clocksource+0x0/0x14 @ 1
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
initcall init_jiffies_clocksource+0x0/0x14 returned 0 after 9765 usecs
calling futex_init+0x0/0xe0 @ 1
futex hash table entries: 256 (order: -1, 3072 bytes, linear)
initcall futex_init+0x0/0xe0 returned 0 after 9765 usecs
calling cgroup_wq_init+0x0/0x38 @ 1
initcall cgroup_wq_init+0x0/0x38 returned 0 after 0 usecs
calling cgroup1_wq_init+0x0/0x38 @ 1
initcall cgroup1_wq_init+0x0/0x38 returned 0 after 0 usecs
calling cpu_pm_init+0x0/0x18 @ 1
initcall cpu_pm_init+0x0/0x18 returned 0 after 0 usecs
calling init_zero_pfn+0x0/0x40 @ 1
initcall init_zero_pfn+0x0/0x40 returned 0 after 0 usecs
calling fsnotify_init+0x0/0x5c @ 1
initcall fsnotify_init+0x0/0x5c returned 0 after 0 usecs
calling filelock_init+0x0/0x78 @ 1
initcall filelock_init+0x0/0x78 returned 0 after 0 usecs
calling init_script_binfmt+0x0/0x1c @ 1
initcall init_script_binfmt+0x0/0x1c returned 0 after 0 usecs
calling init_elf_binfmt+0x0/0x1c @ 1
initcall init_elf_binfmt+0x0/0x1c returned 0 after 0 usecs
calling configfs_init+0x0/0xac @ 1
initcall configfs_init+0x0/0xac returned 0 after 0 usecs
calling debugfs_init+0x0/0x78 @ 1
initcall debugfs_init+0x0/0x78 returned 0 after 0 usecs
calling prandom_init_early+0x0/0xc8 @ 1
initcall prandom_init_early+0x0/0xc8 returned 0 after 0 usecs
calling pinctrl_init+0x0/0xd4 @ 1
pinctrl core: initialized pinctrl subsystem
initcall pinctrl_init+0x0/0xd4 returned 0 after 9765 usecs
calling gpiolib_dev_init+0x0/0xc4 @ 1
initcall gpiolib_dev_init+0x0/0xc4 returned 0 after 0 usecs
calling regulator_init+0x0/0xb0 @ 1
initcall regulator_init+0x0/0xb0 returned 0 after 0 usecs
calling component_debug_init+0x0/0x28 @ 1
initcall component_debug_init+0x0/0x28 returned 0 after 0 usecs
calling soc_bus_register+0x0/0x50 @ 1
initcall soc_bus_register+0x0/0x50 returned 0 after 0 usecs
calling sock_init+0x0/0xa4 @ 1
initcall sock_init+0x0/0xa4 returned 0 after 0 usecs
calling net_inuse_init+0x0/0x24 @ 1
initcall net_inuse_init+0x0/0x24 returned 0 after 0 usecs
calling net_defaults_init+0x0/0x24 @ 1
initcall net_defaults_init+0x0/0x24 returned 0 after 0 usecs
calling init_default_flow_dissectors+0x0/0x4c @ 1
initcall init_default_flow_dissectors+0x0/0x4c returned 0 after 0 usecs
calling netlink_proto_init+0x0/0x164 @ 1
NET: Registered protocol family 16
initcall netlink_proto_init+0x0/0x164 returned 0 after 0 usecs
calling genl_init+0x0/0x38 @ 1
initcall genl_init+0x0/0x38 returned 0 after 0 usecs
calling atomic_pool_init+0x0/0x154 @ 1
DMA: preallocated 256 KiB pool for atomic coherent allocations
initcall atomic_pool_init+0x0/0x154 returned 0 after 9765 usecs
calling irq_sysfs_init+0x0/0xb4 @ 1
initcall irq_sysfs_init+0x0/0xb4 returned 0 after 0 usecs
calling bdi_class_init+0x0/0x5c @ 1
initcall bdi_class_init+0x0/0x5c returned 0 after 0 usecs
calling mm_sysfs_init+0x0/0x3c @ 1
initcall mm_sysfs_init+0x0/0x3c returned 0 after 0 usecs
calling init_per_zone_wmark_min+0x0/0x6c @ 1
initcall init_per_zone_wmark_min+0x0/0x6c returned 0 after 0 usecs
calling mpi_init+0x0/0xa4 @ 1
initcall mpi_init+0x0/0xa4 returned 0 after 0 usecs
calling kobject_uevent_init+0x0/0xc @ 1
initcall kobject_uevent_init+0x0/0xc returned 0 after 0 usecs
calling gpiolib_sysfs_init+0x0/0x7c @ 1
initcall gpiolib_sysfs_init+0x0/0x7c returned 0 after 0 usecs
calling lcd_class_init+0x0/0x58 @ 1
initcall lcd_class_init+0x0/0x58 returned 0 after 0 usecs
calling backlight_class_init+0x0/0xa4 @ 1
initcall backlight_class_init+0x0/0xa4 returned 0 after 0 usecs
calling pmc_register_ops+0x0/0x54 @ 1
initcall pmc_register_ops+0x0/0x54 returned 0 after 0 usecs
calling tty_class_init+0x0/0x40 @ 1
initcall tty_class_init+0x0/0x40 returned 0 after 0 usecs
calling vtconsole_class_init+0x0/0xe8 @ 1
initcall vtconsole_class_init+0x0/0xe8 returned 0 after 0 usecs
calling devlink_class_init+0x0/0x44 @ 1
initcall devlink_class_init+0x0/0x44 returned 0 after 0 usecs
calling software_node_init+0x0/0x40 @ 1
initcall software_node_init+0x0/0x40 returned 0 after 0 usecs
calling wakeup_sources_debugfs_init+0x0/0x38 @ 1
initcall wakeup_sources_debugfs_init+0x0/0x38 returned 0 after 0 usecs
calling wakeup_sources_sysfs_init+0x0/0x30 @ 1
initcall wakeup_sources_sysfs_init+0x0/0x30 returned 0 after 0 usecs
calling regmap_initcall+0x0/0x10 @ 1
initcall regmap_initcall+0x0/0x10 returned 0 after 0 usecs
calling sram_init+0x0/0x10 @ 1
initcall sram_init+0x0/0x10 returned 0 after 0 usecs
calling syscon_init+0x0/0x10 @ 1
initcall syscon_init+0x0/0x10 returned 0 after 0 usecs
calling spi_init+0x0/0x88 @ 1
initcall spi_init+0x0/0x88 returned 0 after 0 usecs
calling i2c_init+0x0/0xc0 @ 1
initcall i2c_init+0x0/0xc0 returned 0 after 0 usecs
calling gate_vma_init+0x0/0x70 @ 1
initcall gate_vma_init+0x0/0x70 returned 0 after 0 usecs
calling customize_machine+0x0/0x30 @ 1
platform f0004000.dma-controller: Linked as a consumer to f0014000.pmc
platform f0010000.dma-controller: Linked as a consumer to f0014000.pmc
platform f000c000.ramc: Linked as a consumer to f0014000.pmc
platform b0000000.sdio-host: Linked as a consumer to f0014000.pmc
platform a0000000.sdio-host: Linked as a consumer to f0014000.pmc
platform 500000.ehci: Linked as a consumer to f0014000.pmc
platform 400000.ohci: Linked as a consumer to f0014000.pmc
platform 300000.gadget: Linked as a consumer to f0014000.pmc
platform f0028000.sha: Linked as a consumer to f0014000.pmc
platform f0028000.sha: Linked as a consumer to f0010000.dma-controller
platform f002c000.aes: Linked as a consumer to f0014000.pmc
platform f002c000.aes: Linked as a consumer to f0010000.dma-controller
platform f8000000.spi: Linked as a consumer to f0014000.pmc
platform f8000000.spi: Linked as a consumer to f0010000.dma-controller
platform f8008000.ethernet: Linked as a consumer to f0014000.pmc
platform f800c000.timer: Linked as a consumer to f0014000.pmc
platform f8010000.timer: Linked as a consumer to f0014000.pmc
platform f8014000.hsmc: Linked as a consumer to f0014000.pmc
platform f8020000.serial: Linked as a consumer to f0014000.pmc
platform f8020000.serial: Linked as a consumer to f0010000.dma-controller
platform f8028000.i2c: Linked as a consumer to f0014000.pmc
platform f8044000.sram: Linked as a consumer to f0014000.pmc
platform f8044000.sram: probe deferral - supplier f0014000.pmc not ready
platform f8048030.timer: Linked as a consumer to f0014000.pmc
platform f8054000.can: Linked as a consumer to f0014000.pmc
platform fc008000.serial: Linked as a consumer to f0014000.pmc
platform fc008000.serial: Linked as a consumer to f0004000.dma-controller
platform fc018000.flexcom: Linked as a consumer to f0014000.pmc
platform fc01c000.trng: Linked as a consumer to f0014000.pmc
platform fc028000.i2c: Linked as a consumer to f0014000.pmc
platform fc030000.adc: Linked as a consumer to f0010000.dma-controller
platform fc030000.adc: Linked as a consumer to f0014000.pmc
platform fc030000.adc: Linked as a consumer to fc038000.pinctrl
platform fc028000.i2c: Linked as a consumer to fc038000.pinctrl
platform fc018000.flexcom: Linked as a sync state only consumer to fc038000.pinctrl
platform fc008000.serial: Linked as a consumer to fc038000.pinctrl
platform f8054000.can: Linked as a consumer to fc038000.pinctrl
platform f8028000.i2c: Linked as a sync state only consumer to fc038000.pinctrl
platform f8028000.i2c: Linked as a consumer to fc038000.pinctrl
platform f8020000.serial: Linked as a consumer to fc038000.pinctrl
platform f8008000.ethernet: Linked as a sync state only consumer to fc038000.pinctrl
platform f8008000.ethernet: Linked as a consumer to fc038000.pinctrl
platform f8000000.spi: Linked as a consumer to fc038000.pinctrl
platform b0000000.sdio-host: Linked as a consumer to fc038000.pinctrl
platform a0000000.sdio-host: Linked as a consumer to fc038000.pinctrl
platform 400000.ohci: Linked as a consumer to fc038000.pinctrl
platform 300000.gadget: Linked as a consumer to fc038000.pinctrl
platform fc038000.pinctrl: Linked as a consumer to f0014000.pmc
platform fc044000.tdes: Linked as a consumer to f0014000.pmc
platform fc044000.tdes: Linked as a consumer to f0010000.dma-controller
platform fc048000.classd: Linked as a consumer to fc038000.pinctrl
platform fc048000.classd: Linked as a consumer to f0014000.pmc
platform fc048000.classd: Linked as a consumer to f0010000.dma-controller
platform fc050000.can: Linked as a consumer to fc038000.pinctrl
platform fc050000.can: Linked as a consumer to f0014000.pmc
platform gpio_keys: Linked as a consumer to fc038000.pinctrl
platform leds: Linked as a consumer to fc038000.pinctrl
AT91: PM: standby: standby, suspend: ulp0
initcall customize_machine+0x0/0x30 returned 0 after 400390 usecs
calling init_atags_procfs+0x0/0xf0 @ 1
No ATAGs?
initcall init_atags_procfs+0x0/0xf0 returned -22 after 0 usecs
calling exceptions_init+0x0/0x94 @ 1
initcall exceptions_init+0x0/0x94 returned 0 after 0 usecs
calling cryptomgr_init+0x0/0xc @ 1
initcall cryptomgr_init+0x0/0xc returned 0 after 0 usecs
calling at91_pinctrl_init+0x0/0x14 @ 1
initcall at91_pinctrl_init+0x0/0x14 returned 0 after 0 usecs
calling dma_bus_init+0x0/0xf0 @ 1
initcall dma_bus_init+0x0/0xf0 returned 0 after 0 usecs
calling dma_channel_table_init+0x0/0x10c @ 1
initcall dma_channel_table_init+0x0/0x10c returned 0 after 0 usecs
calling of_platform_default_populate_init+0x0/0xbc @ 1
initcall of_platform_default_populate_init+0x0/0xbc returned 0 after 0 usecs
calling vfp_kmode_exception_hook_init+0x0/0x3c @ 1
initcall vfp_kmode_exception_hook_init+0x0/0x3c returned 0 after 0 usecs
calling topology_init+0x0/0x28 @ 1
initcall topology_init+0x0/0x28 returned 0 after 0 usecs
calling uid_cache_init+0x0/0x94 @ 1
initcall uid_cache_init+0x0/0x94 returned 0 after 0 usecs
calling param_sysfs_init+0x0/0x208 @ 1
initcall param_sysfs_init+0x0/0x208 returned 0 after 9765 usecs
calling user_namespace_sysctl_init+0x0/0x44 @ 1
initcall user_namespace_sysctl_init+0x0/0x44 returned 0 after 0 usecs
calling crash_save_vmcoreinfo_init+0x0/0x6fc @ 1
initcall crash_save_vmcoreinfo_init+0x0/0x6fc returned 0 after 0 usecs
calling crash_notes_memory_init+0x0/0x40 @ 1
initcall crash_notes_memory_init+0x0/0x40 returned 0 after 0 usecs
calling cgroup_sysfs_init+0x0/0x18 @ 1
initcall cgroup_sysfs_init+0x0/0x18 returned 0 after 0 usecs
calling cgroup_namespaces_init+0x0/0x8 @ 1
initcall cgroup_namespaces_init+0x0/0x8 returned 0 after 0 usecs
calling oom_init+0x0/0x34 @ 1
initcall oom_init+0x0/0x34 returned 0 after 0 usecs
calling default_bdi_init+0x0/0x40 @ 1
initcall default_bdi_init+0x0/0x40 returned 0 after 0 usecs
calling percpu_enable_async+0x0/0x18 @ 1
initcall percpu_enable_async+0x0/0x18 returned 0 after 0 usecs
calling kcompactd_init+0x0/0x60 @ 1
initcall kcompactd_init+0x0/0x60 returned 0 after 0 usecs
calling init_reserve_notifier+0x0/0x8 @ 1
initcall init_reserve_notifier+0x0/0x8 returned 0 after 0 usecs
calling init_admin_reserve+0x0/0x2c @ 1
initcall init_admin_reserve+0x0/0x2c returned 0 after 0 usecs
calling init_user_reserve+0x0/0x2c @ 1
initcall init_user_reserve+0x0/0x2c returned 0 after 0 usecs
calling io_wq_init+0x0/0x50 @ 1
initcall io_wq_init+0x0/0x50 returned 0 after 0 usecs
calling rsa_init+0x0/0x48 @ 1
initcall rsa_init+0x0/0x48 returned 0 after 0 usecs
calling crypto_cmac_module_init+0x0/0xc @ 1
initcall crypto_cmac_module_init+0x0/0xc returned 0 after 0 usecs
calling crypto_null_mod_init+0x0/0x64 @ 1
initcall crypto_null_mod_init+0x0/0x64 returned 0 after 0 usecs
calling sha256_generic_mod_init+0x0/0x10 @ 1
initcall sha256_generic_mod_init+0x0/0x10 returned 0 after 0 usecs
calling crypto_ctr_module_init+0x0/0x10 @ 1
initcall crypto_ctr_module_init+0x0/0x10 returned 0 after 0 usecs
calling crypto_gcm_module_init+0x0/0x68 @ 1
initcall crypto_gcm_module_init+0x0/0x68 returned 0 after 0 usecs
calling crypto_ccm_module_init+0x0/0x10 @ 1
initcall crypto_ccm_module_init+0x0/0x10 returned 0 after 0 usecs
calling aes_init+0x0/0xc @ 1
initcall aes_init+0x0/0xc returned 0 after 0 usecs
calling deflate_mod_init+0x0/0x44 @ 1
initcall deflate_mod_init+0x0/0x44 returned 0 after 0 usecs
calling crc32c_mod_init+0x0/0xc @ 1
initcall crc32c_mod_init+0x0/0xc returned 0 after 0 usecs
calling lzo_mod_init+0x0/0x40 @ 1
initcall lzo_mod_init+0x0/0x40 returned 0 after 0 usecs
calling lzorle_mod_init+0x0/0x40 @ 1
initcall lzorle_mod_init+0x0/0x40 returned 0 after 9765 usecs
calling ghash_mod_init+0x0/0xc @ 1
initcall ghash_mod_init+0x0/0xc returned 0 after 9765 usecs
calling zstd_mod_init+0x0/0x40 @ 1
initcall zstd_mod_init+0x0/0x40 returned 0 after 9765 usecs
calling init_bio+0x0/0xbc @ 1
initcall init_bio+0x0/0xbc returned 0 after 0 usecs
calling blk_settings_init+0x0/0x34 @ 1
initcall blk_settings_init+0x0/0x34 returned 0 after 0 usecs
calling blk_ioc_init+0x0/0x40 @ 1
initcall blk_ioc_init+0x0/0x40 returned 0 after 0 usecs
calling blk_mq_init+0x0/0xb0 @ 1
initcall blk_mq_init+0x0/0xb0 returned 0 after 0 usecs
calling genhd_device_init+0x0/0x6c @ 1
initcall genhd_device_init+0x0/0x6c returned 0 after 0 usecs
calling gpiolib_debugfs_init+0x0/0x38 @ 1
initcall gpiolib_debugfs_init+0x0/0x38 returned 0 after 0 usecs
calling pwm_debugfs_init+0x0/0x38 @ 1
initcall pwm_debugfs_init+0x0/0x38 returned 0 after 0 usecs
calling pwm_sysfs_init+0x0/0x14 @ 1
initcall pwm_sysfs_init+0x0/0x14 returned 0 after 0 usecs
calling fbmem_init+0x0/0xf8 @ 1
initcall fbmem_init+0x0/0xf8 returned 0 after 0 usecs
calling at_dma_init+0x0/0x18 @ 1
initcall at_dma_init+0x0/0x18 returned -19 after 0 usecs
calling at_xdmac_init+0x0/0x18 @ 1
platform f0010000.dma-controller: probe deferral - supplier f0014000.pmc not ready
platform f0004000.dma-controller: probe deferral - supplier f0014000.pmc not ready
initcall at_xdmac_init+0x0/0x18 returned -19 after 19531 usecs
calling atmel_soc_device_init+0x0/0x40 @ 1
AT91: Detected SoC family: sama5d2
AT91: Detected SoC: sama5d27, revision 2
initcall atmel_soc_device_init+0x0/0x40 returned 0 after 0 usecs
calling regulator_fixed_voltage_init+0x0/0x10 @ 1
initcall regulator_fixed_voltage_init+0x0/0x10 returned 0 after 0 usecs
calling misc_init+0x0/0xe0 @ 1
initcall misc_init+0x0/0xe0 returned 0 after 0 usecs
calling act8945a_i2c_init+0x0/0x10 @ 1
initcall act8945a_i2c_init+0x0/0x10 returned 0 after 0 usecs
calling dma_buf_init+0x0/0xb4 @ 1
initcall dma_buf_init+0x0/0xb4 returned 0 after 0 usecs
calling init_scsi+0x0/0x7c @ 1
SCSI subsystem initialized
initcall init_scsi+0x0/0x7c returned 0 after 9765 usecs
calling phy_init+0x0/0x45c @ 1
initcall phy_init+0x0/0x45c returned 0 after 0 usecs
calling usb_common_init+0x0/0x28 @ 1
initcall usb_common_init+0x0/0x28 returned 0 after 0 usecs
calling usb_init+0x0/0x148 @ 1
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
initcall usb_init+0x0/0x148 returned 0 after 19531 usecs
calling usb_udc_init+0x0/0x58 @ 1
initcall usb_udc_init+0x0/0x58 returned 0 after 0 usecs
calling input_init+0x0/0x108 @ 1
initcall input_init+0x0/0x108 returned 0 after 0 usecs
calling rtc_init+0x0/0x58 @ 1
initcall rtc_init+0x0/0x58 returned 0 after 0 usecs
calling at91_twi_init+0x0/0x10 @ 1
platform f8028000.i2c: probe deferral - supplier f0014000.pmc not ready
platform fc028000.i2c: probe deferral - supplier f0014000.pmc not ready
initcall at91_twi_init+0x0/0x10 returned 0 after 19531 usecs
calling i2c_gpio_init+0x0/0x34 @ 1
initcall i2c_gpio_init+0x0/0x34 returned 0 after 0 usecs
calling media_devnode_init+0x0/0x84 @ 1
mc: Linux media interface: v0.10
initcall media_devnode_init+0x0/0x84 returned 0 after 0 usecs
calling videodev_init+0x0/0x84 @ 1
videodev: Linux video capture interface: v2.00
initcall videodev_init+0x0/0x84 returned 0 after 9765 usecs
calling power_supply_class_init+0x0/0x4c @ 1
initcall power_supply_class_init+0x0/0x4c returned 0 after 0 usecs
calling hwmon_init+0x0/0x34 @ 1
initcall hwmon_init+0x0/0x34 returned 0 after 0 usecs
calling mmc_init+0x0/0x38 @ 1
initcall mmc_init+0x0/0x38 returned 0 after 0 usecs
calling leds_init+0x0/0x4c @ 1
initcall leds_init+0x0/0x4c returned 0 after 0 usecs
calling iio_init+0x0/0x8c @ 1
initcall iio_init+0x0/0x8c returned 0 after 0 usecs
calling nvmem_init+0x0/0xc @ 1
initcall nvmem_init+0x0/0xc returned 0 after 0 usecs
calling init_soundcore+0x0/0x40 @ 1
initcall init_soundcore+0x0/0x40 returned 0 after 0 usecs
calling alsa_sound_init+0x0/0xb0 @ 1
Advanced Linux Sound Architecture Driver Initialized.
initcall alsa_sound_init+0x0/0xb0 returned 0 after 0 usecs
calling proto_init+0x0/0xc @ 1
initcall proto_init+0x0/0xc returned 0 after 0 usecs
calling net_dev_init+0x0/0x1c8 @ 1
initcall net_dev_init+0x0/0x1c8 returned 0 after 0 usecs
calling neigh_init+0x0/0xa8 @ 1
initcall neigh_init+0x0/0xa8 returned 0 after 0 usecs
calling fib_notifier_init+0x0/0xc @ 1
initcall fib_notifier_init+0x0/0xc returned 0 after 0 usecs
calling devlink_init+0x0/0x4c @ 1
initcall devlink_init+0x0/0x4c returned 0 after 0 usecs
calling ethnl_init+0x0/0x80 @ 1
initcall ethnl_init+0x0/0x80 returned 0 after 0 usecs
calling nexthop_init+0x0/0xf0 @ 1
initcall nexthop_init+0x0/0xf0 returned 0 after 0 usecs
calling ieee80211_init+0x0/0x28 @ 1
initcall ieee80211_init+0x0/0x28 returned 0 after 0 usecs
calling watchdog_init+0x0/0x80 @ 1
initcall watchdog_init+0x0/0x80 returned 0 after 0 usecs
calling proc_cpu_init+0x0/0x24 @ 1
initcall proc_cpu_init+0x0/0x24 returned 0 after 0 usecs
calling alignment_init+0x0/0xd8 @ 1
initcall alignment_init+0x0/0xd8 returned 0 after 0 usecs
calling iomem_init_inode+0x0/0x84 @ 1
initcall iomem_init_inode+0x0/0x84 returned 0 after 0 usecs
calling clocksource_done_booting+0x0/0x44 @ 1
clocksource: Switched to clocksource timer@f800c000
initcall clocksource_done_booting+0x0/0x44 returned 0 after 4967 usecs
calling init_pipe_fs+0x0/0x54 @ 1
initcall init_pipe_fs+0x0/0x54 returned 0 after 111 usecs
calling inotify_user_setup+0x0/0xc8 @ 1
initcall inotify_user_setup+0x0/0xc8 returned 0 after 33 usecs
calling eventpoll_init+0x0/0xe0 @ 1
initcall eventpoll_init+0x0/0xe0 returned 0 after 64 usecs
calling anon_inode_init+0x0/0x68 @ 1
initcall anon_inode_init+0x0/0x68 returned 0 after 76 usecs
calling proc_locks_init+0x0/0x40 @ 1
initcall proc_locks_init+0x0/0x40 returned 0 after 18 usecs
calling iomap_init+0x0/0x18 @ 1
initcall iomap_init+0x0/0x18 returned 0 after 106 usecs
calling proc_cmdline_init+0x0/0x38 @ 1
initcall proc_cmdline_init+0x0/0x38 returned 0 after 15 usecs
calling proc_consoles_init+0x0/0x3c @ 1
initcall proc_consoles_init+0x0/0x3c returned 0 after 11 usecs
calling proc_cpuinfo_init+0x0/0x28 @ 1
initcall proc_cpuinfo_init+0x0/0x28 returned 0 after 10 usecs
calling proc_devices_init+0x0/0x3c @ 1
initcall proc_devices_init+0x0/0x3c returned 0 after 9 usecs
calling proc_interrupts_init+0x0/0x3c @ 1
initcall proc_interrupts_init+0x0/0x3c returned 0 after 9 usecs
calling proc_loadavg_init+0x0/0x38 @ 1
initcall proc_loadavg_init+0x0/0x38 returned 0 after 9 usecs
calling proc_meminfo_init+0x0/0x38 @ 1
initcall proc_meminfo_init+0x0/0x38 returned 0 after 9 usecs
calling proc_stat_init+0x0/0x28 @ 1
initcall proc_stat_init+0x0/0x28 returned 0 after 8 usecs
calling proc_uptime_init+0x0/0x38 @ 1
initcall proc_uptime_init+0x0/0x38 returned 0 after 9 usecs
calling proc_version_init+0x0/0x38 @ 1
initcall proc_version_init+0x0/0x38 returned 0 after 9 usecs
calling proc_softirqs_init+0x0/0x38 @ 1
initcall proc_softirqs_init+0x0/0x38 returned 0 after 8 usecs
calling proc_kmsg_init+0x0/0x28 @ 1
initcall proc_kmsg_init+0x0/0x28 returned 0 after 8 usecs
calling proc_page_init+0x0/0x44 @ 1
initcall proc_page_init+0x0/0x44 returned 0 after 13 usecs
calling init_ramfs_fs+0x0/0xc @ 1
initcall init_ramfs_fs+0x0/0xc returned 0 after 9 usecs
calling blk_scsi_ioctl_init+0x0/0xe4 @ 1
initcall blk_scsi_ioctl_init+0x0/0xe4 returned 0 after 2 usecs
calling chr_dev_init+0x0/0xc8 @ 1
initcall chr_dev_init+0x0/0xc8 returned 0 after 18587 usecs
calling firmware_class_init+0x0/0xc8 @ 1
initcall firmware_class_init+0x0/0xc8 returned 0 after 10 usecs
calling sysctl_core_init+0x0/0x30 @ 1
initcall sysctl_core_init+0x0/0x30 returned 0 after 93 usecs
calling eth_offload_init+0x0/0x18 @ 1
initcall eth_offload_init+0x0/0x18 returned 0 after 2 usecs
calling inet_init+0x0/0x278 @ 1
NET: Registered protocol family 2
tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
TCP bind hash table entries: 4096 (order: 2, 16384 bytes, linear)
TCP: Hash tables configured (established 4096 bind 4096)
UDP hash table entries: 256 (order: 0, 4096 bytes, linear)
UDP-Lite hash table entries: 256 (order: 0, 4096 bytes, linear)
initcall inet_init+0x0/0x278 returned 0 after 43596 usecs
calling ipv4_offload_init+0x0/0x84 @ 1
initcall ipv4_offload_init+0x0/0x84 returned 0 after 4 usecs
calling af_unix_init+0x0/0x54 @ 1
NET: Registered protocol family 1
initcall af_unix_init+0x0/0x54 returned 0 after 3344 usecs
calling ipv6_offload_init+0x0/0x88 @ 1
initcall ipv6_offload_init+0x0/0x88 returned 0 after 4 usecs
calling init_sunrpc+0x0/0x68 @ 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
initcall init_sunrpc+0x0/0x68 returned 0 after 18911 usecs
calling vlan_offload_init+0x0/0x24 @ 1
initcall vlan_offload_init+0x0/0x24 returned 0 after 3 usecs
calling cfg80211_init+0x0/0xc0 @ 1
initcall cfg80211_init+0x0/0xc0 returned 0 after 1949 usecs
calling populate_rootfs+0x0/0x1d8 @ 1
initcall populate_rootfs+0x0/0x1d8 returned 0 after 439 usecs
calling proc_execdomains_init+0x0/0x38 @ 1
initcall proc_execdomains_init+0x0/0x38 returned 0 after 24 usecs
calling register_warn_debugfs+0x0/0x38 @ 1
initcall register_warn_debugfs+0x0/0x38 returned 0 after 37 usecs
calling ioresources_init+0x0/0x68 @ 1
initcall ioresources_init+0x0/0x68 returned 0 after 23 usecs
calling irq_gc_init_ops+0x0/0x18 @ 1
initcall irq_gc_init_ops+0x0/0x18 returned 0 after 4 usecs
calling irq_pm_init_ops+0x0/0x18 @ 1
initcall irq_pm_init_ops+0x0/0x18 returned 0 after 1 usecs
calling timekeeping_init_ops+0x0/0x18 @ 1
initcall timekeeping_init_ops+0x0/0x18 returned 0 after 1 usecs
calling init_clocksource_sysfs+0x0/0x2c @ 1
initcall init_clocksource_sysfs+0x0/0x2c returned 0 after 377 usecs
calling init_timer_list_procfs+0x0/0x40 @ 1
random: fast init done
initcall init_timer_list_procfs+0x0/0x40 returned 0 after 18 usecs
calling alarmtimer_init+0x0/0xb8 @ 1
initcall alarmtimer_init+0x0/0xb8 returned 0 after 200 usecs
calling init_posix_timers+0x0/0x40 @ 1
initcall init_posix_timers+0x0/0x40 returned 0 after 86 usecs
calling clockevents_init_sysfs+0x0/0x6c @ 1
initcall clockevents_init_sysfs+0x0/0x6c returned 0 after 342 usecs
calling sched_clock_syscore_init+0x0/0x18 @ 1
initcall sched_clock_syscore_init+0x0/0x18 returned 0 after 2 usecs
calling proc_modules_init+0x0/0x28 @ 1
initcall proc_modules_init+0x0/0x28 returned 0 after 21 usecs
calling kallsyms_init+0x0/0x28 @ 1
initcall kallsyms_init+0x0/0x28 returned 0 after 12 usecs
calling seccomp_sysctl_init+0x0/0x30 @ 1
initcall seccomp_sysctl_init+0x0/0x30 returned 0 after 36 usecs
calling utsname_sysctl_init+0x0/0x18 @ 1
initcall utsname_sysctl_init+0x0/0x18 returned 0 after 26 usecs
calling system_trusted_keyring_init+0x0/0x88 @ 1
Initialise system trusted keyrings
initcall system_trusted_keyring_init+0x0/0x88 returned 0 after 3419 usecs
calling kswapd_init+0x0/0x18 @ 1
initcall kswapd_init+0x0/0x18 returned 0 after 327 usecs
calling extfrag_debug_init+0x0/0x70 @ 1
initcall extfrag_debug_init+0x0/0x70 returned 0 after 66 usecs
calling slab_proc_init+0x0/0x28 @ 1
initcall slab_proc_init+0x0/0x28 returned 0 after 21 usecs
calling workingset_init+0x0/0x9c @ 1
workingset: timestamp_bits=30 max_order=17 bucket_order=0
initcall workingset_init+0x0/0x9c returned 0 after 5523 usecs
calling proc_vmalloc_init+0x0/0x3c @ 1
initcall proc_vmalloc_init+0x0/0x3c returned 0 after 12 usecs
calling memblock_init_debugfs+0x0/0x70 @ 1
initcall memblock_init_debugfs+0x0/0x70 returned 0 after 44 usecs
calling cpucache_init+0x0/0x60 @ 1
initcall cpucache_init+0x0/0x60 returned 0 after 14 usecs
calling fcntl_init+0x0/0x40 @ 1
initcall fcntl_init+0x0/0x40 returned 0 after 29 usecs
calling proc_filesystems_init+0x0/0x38 @ 1
initcall proc_filesystems_init+0x0/0x38 returned 0 after 15 usecs
calling start_dirtytime_writeback+0x0/0x34 @ 1
initcall start_dirtytime_writeback+0x0/0x34 returned 0 after 6 usecs
calling blkdev_init+0x0/0x18 @ 1
initcall blkdev_init+0x0/0x18 returned 0 after 59 usecs
calling dio_init+0x0/0x40 @ 1
initcall dio_init+0x0/0x40 returned 0 after 60 usecs
calling dnotify_init+0x0/0x84 @ 1
initcall dnotify_init+0x0/0x84 returned 0 after 35 usecs
calling fanotify_user_setup+0x0/0x80 @ 1
initcall fanotify_user_setup+0x0/0x80 returned 0 after 19 usecs
calling aio_setup+0x0/0x84 @ 1
initcall aio_setup+0x0/0x84 returned 0 after 161 usecs
calling io_uring_init+0x0/0x40 @ 1
initcall io_uring_init+0x0/0x40 returned 0 after 10 usecs
calling mbcache_init+0x0/0x48 @ 1
initcall mbcache_init+0x0/0x48 returned 0 after 56 usecs
calling init_grace+0x0/0xc @ 1
initcall init_grace+0x0/0xc returned 0 after 13 usecs
calling init_devpts_fs+0x0/0x2c @ 1
initcall init_devpts_fs+0x0/0x2c returned 0 after 56 usecs
calling ext4_init_fs+0x0/0x1c0 @ 1
initcall ext4_init_fs+0x0/0x1c0 returned 0 after 333 usecs
calling journal_init+0x0/0x13c @ 1
initcall journal_init+0x0/0x13c returned 0 after 127 usecs
calling init_fat_fs+0x0/0x60 @ 1
initcall init_fat_fs+0x0/0x60 returned 0 after 38 usecs
calling init_vfat_fs+0x0/0xc @ 1
initcall init_vfat_fs+0x0/0xc returned 0 after 8 usecs
calling init_nfs_fs+0x0/0x150 @ 1
initcall init_nfs_fs+0x0/0x150 returned 0 after 765 usecs
calling init_nfs_v2+0x0/0x18 @ 1
initcall init_nfs_v2+0x0/0x18 returned 0 after 1 usecs
calling init_nfs_v3+0x0/0x18 @ 1
initcall init_nfs_v3+0x0/0x18 returned 0 after 1 usecs
calling init_nlm+0x0/0x64 @ 1
initcall init_nlm+0x0/0x64 returned 0 after 59 usecs
calling init_nls_cp437+0x0/0x10 @ 1
initcall init_nls_cp437+0x0/0x10 returned 0 after 1 usecs
calling init_nls_cp850+0x0/0x10 @ 1
initcall init_nls_cp850+0x0/0x10 returned 0 after 0 usecs
calling init_nls_iso8859_1+0x0/0x10 @ 1
initcall init_nls_iso8859_1+0x0/0x10 returned 0 after 0 usecs
calling init_nls_utf8+0x0/0x2c @ 1
initcall init_nls_utf8+0x0/0x2c returned 0 after 3 usecs
calling ipc_init+0x0/0x28 @ 1
initcall ipc_init+0x0/0x28 returned 0 after 78 usecs
calling ipc_sysctl_init+0x0/0x18 @ 1
initcall ipc_sysctl_init+0x0/0x18 returned 0 after 41 usecs
calling key_proc_init+0x0/0x88 @ 1
initcall key_proc_init+0x0/0x88 returned 0 after 18 usecs
calling crypto_algapi_init+0x0/0x10 @ 1
initcall crypto_algapi_init+0x0/0x10 returned 0 after 11 usecs
calling asymmetric_key_init+0x0/0xc @ 1
Key type asymmetric registered
initcall asymmetric_key_init+0x0/0xc returned 0 after 3030 usecs
calling x509_key_init+0x0/0xc @ 1
Asymmetric key parser 'x509' registered
initcall x509_key_init+0x0/0xc returned 0 after 3835 usecs
calling proc_genhd_init+0x0/0x60 @ 1
initcall proc_genhd_init+0x0/0x60 returned 0 after 18 usecs
calling deadline_init+0x0/0xc @ 1
io scheduler mq-deadline registered
initcall deadline_init+0x0/0xc returned 0 after 3463 usecs
calling kyber_init+0x0/0xc @ 1
io scheduler kyber registered
initcall kyber_init+0x0/0xc returned 0 after 2905 usecs
calling sg_pool_init+0x0/0xe8 @ 1
initcall sg_pool_init+0x0/0xe8 returned 0 after 138 usecs
calling atmel_pinctrl_driver_init+0x0/0x10 @ 1
platform fc038000.pinctrl: probe deferral - supplier f0014000.pmc not ready
initcall atmel_pinctrl_driver_init+0x0/0x10 returned 0 after 7697 usecs
calling atmel_pwm_driver_init+0x0/0x10 @ 1
initcall atmel_pwm_driver_init+0x0/0x10 returned 0 after 371 usecs
calling atmel_hlcdc_pwm_driver_init+0x0/0x10 @ 1
initcall atmel_hlcdc_pwm_driver_init+0x0/0x10 returned 0 after 209 usecs
calling atmel_tcb_pwm_driver_init+0x0/0x10 @ 1
initcall atmel_tcb_pwm_driver_init+0x0/0x10 returned 0 after 192 usecs
calling pwm_backlight_driver_init+0x0/0x10 @ 1
initcall pwm_backlight_driver_init+0x0/0x10 returned 0 after 178 usecs
calling of_fixed_factor_clk_driver_init+0x0/0x10 @ 1
initcall of_fixed_factor_clk_driver_init+0x0/0x10 returned 0 after 176 usecs
calling of_fixed_clk_driver_init+0x0/0x10 @ 1
initcall of_fixed_clk_driver_init+0x0/0x10 returned 0 after 194 usecs
calling gpio_clk_driver_init+0x0/0x10 @ 1
initcall gpio_clk_driver_init+0x0/0x10 returned 0 after 222 usecs
calling act8865_pmic_driver_init+0x0/0x10 @ 1
initcall act8865_pmic_driver_init+0x0/0x10 returned 0 after 84 usecs
calling act8945a_pmic_driver_init+0x0/0x10 @ 1
initcall act8945a_pmic_driver_init+0x0/0x10 returned 0 after 120 usecs
calling n_null_init+0x0/0x20 @ 1
initcall n_null_init+0x0/0x20 returned 0 after 1 usecs
calling pty_init+0x0/0x3cc @ 1
initcall pty_init+0x0/0x3cc returned 0 after 2349 usecs
calling atmel_serial_init+0x0/0x44 @ 1
initcall atmel_serial_init+0x0/0x44 returned 0 after 374 usecs
calling hwrng_modinit+0x0/0x84 @ 1
initcall hwrng_modinit+0x0/0x84 returned 0 after 380 usecs
calling atmel_trng_driver_init+0x0/0x10 @ 1
platform fc01c000.trng: probe deferral - supplier f0014000.pmc not ready
initcall atmel_trng_driver_init+0x0/0x10 returned 0 after 7359 usecs
calling drm_kms_helper_init+0x0/0x10 @ 1
initcall drm_kms_helper_init+0x0/0x10 returned 0 after 0 usecs
calling drm_core_init+0x0/0xc0 @ 1
initcall drm_core_init+0x0/0xc0 returned 0 after 136 usecs
calling atmel_hlcdc_dc_platform_driver_init+0x0/0x10 @ 1
initcall atmel_hlcdc_dc_platform_driver_init+0x0/0x10 returned 0 after 251 usecs
calling panel_simple_init+0x0/0x1c @ 1
initcall panel_simple_init+0x0/0x1c returned 0 after 4764 usecs
calling topology_sysfs_init+0x0/0x40 @ 1
initcall topology_sysfs_init+0x0/0x40 returned 0 after 83 usecs
calling cacheinfo_sysfs_init+0x0/0x40 @ 1
initcall cacheinfo_sysfs_init+0x0/0x40 returned -2 after 4 usecs
calling devcoredump_init+0x0/0x14 @ 1
initcall devcoredump_init+0x0/0x14 returned 0 after 72 usecs
calling brd_init+0x0/0x1c0 @ 1
brd: module loaded
initcall brd_init+0x0/0x1c0 returned 0 after 7717 usecs
calling loop_init+0x0/0x124 @ 1
loop: module loaded
initcall loop_init+0x0/0x124 returned 0 after 20573 usecs
calling ssc_driver_init+0x0/0x10 @ 1
initcall ssc_driver_init+0x0/0x10 returned 0 after 470 usecs
calling at24_init+0x0/0x48 @ 1
initcall at24_init+0x0/0x48 returned 0 after 144 usecs
calling at91_usart_mfd_init+0x0/0x10 @ 1
platform f8020000.serial: probe deferral - supplier f0014000.pmc not ready
platform fc008000.serial: probe deferral - supplier f0014000.pmc not ready
initcall at91_usart_mfd_init+0x0/0x10 returned 0 after 14660 usecs
calling atmel_flexcom_driver_init+0x0/0x10 @ 1
platform fc018000.flexcom: probe deferral - supplier f0014000.pmc not ready
initcall atmel_flexcom_driver_init+0x0/0x10 returned 0 after 7504 usecs
calling atmel_hlcdc_driver_init+0x0/0x10 @ 1
initcall atmel_hlcdc_driver_init+0x0/0x10 returned 0 after 449 usecs
calling init_sd+0x0/0x194 @ 1
initcall init_sd+0x0/0x194 returned 0 after 174 usecs
calling init_mtd+0x0/0x118 @ 1
initcall init_mtd+0x0/0x118 returned 0 after 357 usecs
calling cmdline_parser_init+0x0/0x30 @ 1
initcall cmdline_parser_init+0x0/0x30 returned 0 after 1 usecs
calling ofpart_parser_init+0x0/0x2c @ 1
initcall ofpart_parser_init+0x0/0x2c returned 0 after 0 usecs
calling init_mtdblock+0x0/0xc @ 1
initcall init_mtdblock+0x0/0xc returned 0 after 14 usecs
calling cfi_probe_init+0x0/0x18 @ 1
initcall cfi_probe_init+0x0/0x18 returned 0 after 1 usecs
calling atmel_nand_controller_driver_init+0x0/0x10 @ 1
initcall atmel_nand_controller_driver_init+0x0/0x10 returned 0 after 681 usecs
calling atmel_pmecc_driver_init+0x0/0x10 @ 1
initcall atmel_pmecc_driver_init+0x0/0x10 returned 0 after 669 usecs
calling spi_nor_driver_init+0x0/0x10 @ 1
initcall spi_nor_driver_init+0x0/0x10 returned 0 after 82 usecs
calling ubi_gluebi_init+0x0/0x10 @ 1
initcall ubi_gluebi_init+0x0/0x10 returned 0 after 7 usecs
calling atmel_spi_driver_init+0x0/0x10 @ 1
platform f8000000.spi: probe deferral - supplier f0014000.pmc not ready
initcall atmel_spi_driver_init+0x0/0x10 returned 0 after 7165 usecs
calling spi_gpio_driver_init+0x0/0x10 @ 1
initcall spi_gpio_driver_init+0x0/0x10 returned 0 after 210 usecs
calling net_olddevs_init+0x0/0x74 @ 1
initcall net_olddevs_init+0x0/0x74 returned 0 after 27 usecs
calling blackhole_netdev_init+0x0/0x7c @ 1
initcall blackhole_netdev_init+0x0/0x7c returned 0 after 77 usecs
calling fixed_mdio_bus_init+0x0/0x114 @ 1
libphy: Fixed MDIO Bus: probed
initcall fixed_mdio_bus_init+0x0/0x114 returned 0 after 4129 usecs
calling phy_module_init+0x0/0x14 @ 1
initcall phy_module_init+0x0/0x14 returned 0 after 854 usecs
calling can_dev_init+0x0/0x24 @ 1
CAN device driver interface
initcall can_dev_init+0x0/0x24 returned 0 after 2718 usecs
calling at91_can_driver_init+0x0/0x10 @ 1
initcall at91_can_driver_init+0x0/0x10 returned 0 after 362 usecs
calling m_can_plat_driver_init+0x0/0x10 @ 1
platform f8054000.can: probe deferral - supplier f0014000.pmc not ready
platform fc050000.can: probe deferral - supplier fc038000.pinctrl not ready
initcall m_can_plat_driver_init+0x0/0x10 returned 0 after 14415 usecs
calling macb_driver_init+0x0/0x10 @ 1
platform f8008000.ethernet: probe deferral - supplier f0014000.pmc not ready
initcall macb_driver_init+0x0/0x10 returned 0 after 8251 usecs
calling ehci_hcd_init+0x0/0x7c @ 1
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
initcall ehci_hcd_init+0x0/0x7c returned 0 after 5617 usecs
calling ehci_atmel_init+0x0/0x54 @ 1
ehci-atmel: EHCI Atmel driver
platform 500000.ehci: probe deferral - supplier f0014000.pmc not ready
initcall ehci_atmel_init+0x0/0x54 returned 0 after 9946 usecs
calling ohci_hcd_mod_init+0x0/0x74 @ 1
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
initcall ohci_hcd_mod_init+0x0/0x74 returned 0 after 5314 usecs
calling ohci_at91_init+0x0/0x6c @ 1
ohci-atmel: OHCI Atmel driver
platform 400000.ohci: probe deferral - supplier f0014000.pmc not ready
initcall ohci_at91_init+0x0/0x6c returned 0 after 9979 usecs
calling acm_init+0x0/0x114 @ 1
usbcore: registered new interface driver cdc_acm
cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
initcall acm_init+0x0/0x114 returned 0 after 12116 usecs
calling usb_storage_driver_init+0x0/0x38 @ 1
usbcore: registered new interface driver usb-storage
initcall usb_storage_driver_init+0x0/0x38 returned 0 after 5225 usecs
calling usb_serial_init+0x0/0x174 @ 1
usbcore: registered new interface driver usbserial_generic
usbserial: USB Serial support registered for generic
initcall usb_serial_init+0x0/0x174 returned 0 after 11010 usecs
calling usb_serial_module_init+0x0/0x1c @ 1
usbcore: registered new interface driver ftdi_sio
usbserial: USB Serial support registered for FTDI USB Serial Device
initcall usb_serial_module_init+0x0/0x1c returned 0 after 11404 usecs
calling usb_serial_module_init+0x0/0x18 @ 1
usbcore: registered new interface driver pl2303
usbserial: USB Serial support registered for pl2303
initcall usb_serial_module_init+0x0/0x18 returned 0 after 9743 usecs
calling gadget_cfs_init+0x0/0x20 @ 1
initcall gadget_cfs_init+0x0/0x20 returned 0 after 158 usecs
calling udc_driver_init+0x0/0x18 @ 1
platform 300000.gadget: probe deferral - supplier f0014000.pmc not ready
initcall udc_driver_init+0x0/0x18 returned -19 after 7524 usecs
calling acmmod_init+0x0/0xc @ 1
initcall acmmod_init+0x0/0xc returned 0 after 3 usecs
calling userial_init+0x0/0x140 @ 1
initcall userial_init+0x0/0x140 returned 0 after 18 usecs
calling gsermod_init+0x0/0xc @ 1
initcall gsermod_init+0x0/0xc returned 0 after 2 usecs
calling obexmod_init+0x0/0xc @ 1
initcall obexmod_init+0x0/0xc returned 0 after 2 usecs
calling init+0x0/0xd0 @ 1
udc-core: couldn't find an available UDC - added [g_serial] to list of pending drivers
initcall init+0x0/0xd0 returned 0 after 8253 usecs
calling input_leds_init+0x0/0xc @ 1
initcall input_leds_init+0x0/0xc returned 0 after 5 usecs
calling evdev_init+0x0/0xc @ 1
initcall evdev_init+0x0/0xc returned 0 after 2 usecs
calling qt1070_driver_init+0x0/0x10 @ 1
initcall qt1070_driver_init+0x0/0x10 returned 0 after 95 usecs
calling grts_driver_init+0x0/0x10 @ 1
initcall grts_driver_init+0x0/0x10 returned 0 after 240 usecs
calling mxt_driver_init+0x0/0x10 @ 1
initcall mxt_driver_init+0x0/0x10 returned 0 after 78 usecs
calling at91_rtc_driver_init+0x0/0x18 @ 1
at91_rtc f80480b0.rtc: registered as rtc0
at91_rtc f80480b0.rtc: setting system clock to 2018-03-13T04:35:46 UTC (1520915746)
at91_rtc f80480b0.rtc: AT91 Real Time Clock driver.
initcall at91_rtc_driver_init+0x0/0x18 returned 0 after 19476 usecs
calling i2c_dev_init+0x0/0xb8 @ 1
i2c /dev entries driver
initcall i2c_dev_init+0x0/0xb8 returned 0 after 2416 usecs
calling atmel_isi_driver_init+0x0/0x10 @ 1
initcall atmel_isi_driver_init+0x0/0x10 returned 0 after 302 usecs
calling atmel_isc_driver_init+0x0/0x10 @ 1
initcall atmel_isc_driver_init+0x0/0x10 returned 0 after 231 usecs
calling at91_poweroff_driver_init+0x0/0x18 @ 1
initcall at91_poweroff_driver_init+0x0/0x18 returned -19 after 361 usecs
calling at91_reset_driver_init+0x0/0x18 @ 1
at91-reset f8048000.rstc: Starting after wakeup
initcall at91_reset_driver_init+0x0/0x18 returned 0 after 5364 usecs
calling at91_shdwc_driver_init+0x0/0x18 @ 1
initcall at91_shdwc_driver_init+0x0/0x18 returned 0 after 1121 usecs
calling act8945a_charger_driver_init+0x0/0x10 @ 1
initcall act8945a_charger_driver_init+0x0/0x10 returned 0 after 176 usecs
calling at91wdt_driver_init+0x0/0x18 @ 1
initcall at91wdt_driver_init+0x0/0x18 returned -19 after 290 usecs
calling sama5d4_wdt_driver_init+0x0/0x10 @ 1
sama5d4_wdt f8048040.watchdog: initialized (timeout = 16 sec, nowayout = 0)
initcall sama5d4_wdt_driver_init+0x0/0x10 returned 0 after 8682 usecs
calling mmc_pwrseq_simple_driver_init+0x0/0x10 @ 1
initcall mmc_pwrseq_simple_driver_init+0x0/0x10 returned 0 after 266 usecs
calling mmc_pwrseq_emmc_driver_init+0x0/0x10 @ 1
initcall mmc_pwrseq_emmc_driver_init+0x0/0x10 returned 0 after 219 usecs
calling mmc_blk_init+0x0/0x104 @ 1
initcall mmc_blk_init+0x0/0x104 returned 0 after 180 usecs
calling sdhci_drv_init+0x0/0x24 @ 1
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
initcall sdhci_drv_init+0x0/0x24 returned 0 after 8530 usecs
calling atmci_driver_init+0x0/0x10 @ 1
initcall atmci_driver_init+0x0/0x10 returned 0 after 230 usecs
calling sdhci_pltfm_drv_init+0x0/0x18 @ 1
sdhci-pltfm: SDHCI platform and OF driver helper
initcall sdhci_pltfm_drv_init+0x0/0x18 returned 0 after 4701 usecs
calling sdhci_at91_driver_init+0x0/0x10 @ 1
initcall sdhci_at91_driver_init+0x0/0x10 returned 0 after 294 usecs
calling gpio_led_driver_init+0x0/0x10 @ 1
platform leds: probe deferral - supplier fc038000.pinctrl not ready
initcall gpio_led_driver_init+0x0/0x10 returned 0 after 6758 usecs
calling led_pwm_driver_init+0x0/0x10 @ 1
initcall led_pwm_driver_init+0x0/0x10 returned 0 after 208 usecs
calling timer_led_trigger_init+0x0/0xc @ 1
initcall timer_led_trigger_init+0x0/0xc returned 0 after 9 usecs
calling heartbeat_trig_init+0x0/0x40 @ 1
initcall heartbeat_trig_init+0x0/0x40 returned 0 after 7 usecs
calling gpio_led_trigger_init+0x0/0xc @ 1
initcall gpio_led_trigger_init+0x0/0xc returned 0 after 3 usecs
calling atmel_aes_driver_init+0x0/0x10 @ 1
platform f002c000.aes: probe deferral - supplier f0014000.pmc not ready
initcall atmel_aes_driver_init+0x0/0x10 returned 0 after 7158 usecs
calling atmel_sha_driver_init+0x0/0x10 @ 1
platform f0028000.sha: probe deferral - supplier f0014000.pmc not ready
initcall atmel_sha_driver_init+0x0/0x10 returned 0 after 7145 usecs
calling atmel_tdes_driver_init+0x0/0x10 @ 1
platform fc044000.tdes: probe deferral - supplier f0014000.pmc not ready
initcall atmel_tdes_driver_init+0x0/0x10 returned 0 after 7176 usecs
calling hid_init+0x0/0x54 @ 1
initcall hid_init+0x0/0x54 returned 0 after 159 usecs
calling hid_init+0x0/0x60 @ 1
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
initcall hid_init+0x0/0x60 returned 0 after 7556 usecs
calling atmel_ramc_driver_init+0x0/0x10 @ 1
platform f000c000.ramc: probe deferral - supplier f0014000.pmc not ready
initcall atmel_ramc_driver_init+0x0/0x10 returned 0 after 7447 usecs
calling atmel_ebi_driver_init+0x0/0x18 @ 1
initcall atmel_ebi_driver_init+0x0/0x18 returned -19 after 665 usecs
calling at91_adc_driver_init+0x0/0x10 @ 1
initcall at91_adc_driver_init+0x0/0x10 returned 0 after 426 usecs
calling at91_adc_driver_init+0x0/0x10 @ 1
platform fc030000.adc: probe deferral - wait for supplier pmic@5b
initcall at91_adc_driver_init+0x0/0x10 returned 0 after 6591 usecs
calling alsa_timer_init+0x0/0x238 @ 1
platform a0000000.sdio-host: probe deferral - supplier f0014000.pmc not ready
platform b0000000.sdio-host: probe deferral - wait for supplier pmic@5b
initcall alsa_timer_init+0x0/0x238 returned 0 after 15501 usecs
calling alsa_pcm_init+0x0/0x6c @ 1
initcall alsa_pcm_init+0x0/0x6c returned 0 after 19 usecs
calling snd_soc_init+0x0/0x8c @ 1
initcall snd_soc_init+0x0/0x8c returned 0 after 909 usecs
calling wm8904_i2c_driver_init+0x0/0x10 @ 1
initcall wm8904_i2c_driver_init+0x0/0x10 returned 0 after 87 usecs
calling atmel_i2s_driver_init+0x0/0x10 @ 1
initcall atmel_i2s_driver_init+0x0/0x10 returned 0 after 273 usecs
calling atmel_asoc_wm8904_driver_init+0x0/0x10 @ 1
initcall atmel_asoc_wm8904_driver_init+0x0/0x10 returned 0 after 237 usecs
calling atmel_classd_driver_init+0x0/0x10 @ 1
platform fc048000.classd: probe deferral - supplier fc038000.pinctrl not ready
initcall atmel_classd_driver_init+0x0/0x10 returned 0 after 7821 usecs
calling atmel_pdmic_driver_init+0x0/0x10 @ 1
initcall atmel_pdmic_driver_init+0x0/0x10 returned 0 after 231 usecs
calling sock_diag_init+0x0/0x40 @ 1
initcall sock_diag_init+0x0/0x40 returned 0 after 102 usecs
calling gre_offload_init+0x0/0x4c @ 1
initcall gre_offload_init+0x0/0x4c returned 0 after 3 usecs
calling sysctl_ipv4_init+0x0/0x54 @ 1
initcall sysctl_ipv4_init+0x0/0x54 returned 0 after 348 usecs
calling tunnel4_init+0x0/0x5c @ 1
initcall tunnel4_init+0x0/0x5c returned 0 after 2 usecs
calling cubictcp_register+0x0/0x6c @ 1
initcall cubictcp_register+0x0/0x6c returned 0 after 7 usecs
calling inet6_init+0x0/0x31c @ 1
NET: Registered protocol family 10
Segment Routing with IPv6
initcall inet6_init+0x0/0x31c returned 0 after 7986 usecs
calling sit_init+0x0/0xc0 @ 1
sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
initcall sit_init+0x0/0xc0 returned 0 after 6243 usecs
calling packet_init+0x0/0x78 @ 1
NET: Registered protocol family 17
initcall packet_init+0x0/0x78 returned 0 after 3441 usecs
calling can_init+0x0/0xbc @ 1
can: controller area network core
NET: Registered protocol family 29
initcall can_init+0x0/0xbc returned 0 after 6811 usecs
calling raw_module_init+0x0/0x38 @ 1
can: raw protocol
initcall raw_module_init+0x0/0x38 returned 0 after 1782 usecs
calling bcm_module_init+0x0/0x4c @ 1
can: broadcast manager protocol
initcall bcm_module_init+0x0/0x4c returned 0 after 3118 usecs
calling cgw_module_init+0x0/0x158 @ 1
can: netlink gateway - max_hops=1
initcall cgw_module_init+0x0/0x158 returned 0 after 3374 usecs
calling init_machine_late+0x0/0x8c @ 1
initcall init_machine_late+0x0/0x8c returned 0 after 19 usecs
calling init_oops_id+0x0/0x40 @ 1
initcall init_oops_id+0x0/0x40 returned 0 after 12 usecs
calling reboot_ksysfs_init+0x0/0x4c @ 1
initcall reboot_ksysfs_init+0x0/0x4c returned 0 after 55 usecs
calling pm_debugfs_init+0x0/0x38 @ 1
initcall pm_debugfs_init+0x0/0x38 returned 0 after 83 usecs
calling printk_late_init+0x0/0x1d4 @ 1
initcall printk_late_init+0x0/0x1d4 returned 0 after 8 usecs
calling tk_debug_sleep_time_init+0x0/0x38 @ 1
initcall tk_debug_sleep_time_init+0x0/0x38 returned 0 after 24 usecs
calling load_system_certificate_list+0x0/0x104 @ 1
Loading compiled-in X.509 certificates
initcall load_system_certificate_list+0x0/0x104 returned 0 after 3736 usecs
calling fault_around_debugfs+0x0/0x38 @ 1
initcall fault_around_debugfs+0x0/0x38 returned 0 after 21 usecs
calling check_early_ioremap_leak+0x0/0x70 @ 1
initcall check_early_ioremap_leak+0x0/0x70 returned 0 after 1 usecs
calling ubifs_init+0x0/0xcc @ 1
initcall ubifs_init+0x0/0xcc returned 0 after 5487 usecs
calling init_root_keyring+0x0/0xc @ 1
initcall init_root_keyring+0x0/0xc returned 0 after 132 usecs
calling blk_timeout_init+0x0/0x18 @ 1
initcall blk_timeout_init+0x0/0x18 returned 0 after 1 usecs
calling prandom_init_late+0x0/0x38 @ 1
initcall prandom_init_late+0x0/0x38 returned 0 after 3 usecs
calling clk_debug_init+0x0/0x114 @ 1
initcall clk_debug_init+0x0/0x114 returned 0 after 10307 usecs
calling sync_state_resume_initcall+0x0/0x10 @ 1
initcall sync_state_resume_initcall+0x0/0x10 returned 0 after 4 usecs
calling deferred_probe_initcall+0x0/0xac @ 1
platform f8044000.sram: probe deferral - supplier f0014000.pmc not ready
platform f8028000.i2c: probe deferral - supplier f0014000.pmc not ready
platform fc028000.i2c: probe deferral - supplier f0014000.pmc not ready
platform fc038000.pinctrl: probe deferral - supplier f0014000.pmc not ready
platform fc01c000.trng: probe deferral - supplier f0014000.pmc not ready
platform f8020000.serial: probe deferral - supplier f0014000.pmc not ready
platform fc008000.serial: probe deferral - supplier f0014000.pmc not ready
platform fc018000.flexcom: probe deferral - supplier f0014000.pmc not ready
platform f8000000.spi: probe deferral - supplier f0014000.pmc not ready
platform f8054000.can: probe deferral - supplier f0014000.pmc not ready
platform fc050000.can: probe deferral - supplier fc038000.pinctrl not ready
platform f8008000.ethernet: probe deferral - supplier f0014000.pmc not ready
platform 500000.ehci: probe deferral - supplier f0014000.pmc not ready
platform 400000.ohci: probe deferral - supplier f0014000.pmc not ready
platform leds: probe deferral - supplier fc038000.pinctrl not ready
platform f002c000.aes: probe deferral - supplier f0014000.pmc not ready
platform f0028000.sha: probe deferral - supplier f0014000.pmc not ready
platform fc044000.tdes: probe deferral - supplier f0014000.pmc not ready
platform f000c000.ramc: probe deferral - supplier f0014000.pmc not ready
platform fc030000.adc: probe deferral - wait for supplier pmic@5b
platform fc048000.classd: probe deferral - supplier fc038000.pinctrl not ready
platform a0000000.sdio-host: probe deferral - supplier f0014000.pmc not ready
platform b0000000.sdio-host: probe deferral - wait for supplier pmic@5b
platform f8044000.sram: probe deferral - supplier f0014000.pmc not ready
platform f8028000.i2c: probe deferral - supplier f0014000.pmc not ready
platform fc028000.i2c: probe deferral - supplier f0014000.pmc not ready
platform fc038000.pinctrl: probe deferral - supplier f0014000.pmc not ready
platform fc01c000.trng: probe deferral - supplier f0014000.pmc not ready
platform f8020000.serial: probe deferral - supplier f0014000.pmc not ready
platform fc008000.serial: probe deferral - supplier f0014000.pmc not ready
platform fc018000.flexcom: probe deferral - supplier f0014000.pmc not ready
platform f8000000.spi: probe deferral - supplier f0014000.pmc not ready
platform f8054000.can: probe deferral - supplier f0014000.pmc not ready
platform fc050000.can: probe deferral - supplier fc038000.pinctrl not ready
platform f8008000.ethernet: probe deferral - supplier f0014000.pmc not ready
platform 500000.ehci: probe deferral - supplier f0014000.pmc not ready
platform 400000.ohci: probe deferral - supplier f0014000.pmc not ready
platform leds: probe deferral - supplier fc038000.pinctrl not ready
platform f002c000.aes: probe deferral - supplier f0014000.pmc not ready
platform f0028000.sha: probe deferral - supplier f0014000.pmc not ready
platform fc044000.tdes: probe deferral - supplier f0014000.pmc not ready
platform f000c000.ramc: probe deferral - supplier f0014000.pmc not ready
platform fc030000.adc: probe deferral - wait for supplier pmic@5b
platform fc048000.classd: probe deferral - supplier fc038000.pinctrl not ready
platform a0000000.sdio-host: probe deferral - supplier f0014000.pmc not ready
platform b0000000.sdio-host: probe deferral - wait for supplier pmic@5b
initcall deferred_probe_initcall+0x0/0xac returned 0 after 334923 usecs
calling ubi_init+0x0/0x218 @ 1
initcall ubi_init+0x0/0x218 returned 0 after 509 usecs
calling gpio_keys_init+0x0/0x10 @ 1
platform gpio_keys: probe deferral - supplier fc038000.pinctrl not ready
initcall gpio_keys_init+0x0/0x10 returned 0 after 7382 usecs
calling of_fdt_raw_init+0x0/0x80 @ 1
initcall of_fdt_raw_init+0x0/0x80 returned 0 after 342 usecs
calling tcp_congestion_default+0x0/0x14 @ 1
initcall tcp_congestion_default+0x0/0x14 returned 0 after 5 usecs
calling ip_auto_config+0x0/0x11d8 @ 1
initcall ip_auto_config+0x0/0x11d8 returned -19 after 12175999 usecs
calling regulatory_init_db+0x0/0x1f4 @ 1
cfg80211: Loading compiled-in X.509 certificates for regulatory database
cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
initcall regulatory_init_db+0x0/0x1f4 returned 0 after 19741 usecs
calling clk_disable_unused+0x0/0x108 @ 1

#45 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 46 lines · On Mon, Jan 25, 2021 at 9:05 AM <[email protected]> wrote:

Hi, Saravana,

Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

next-20210125 fails to boot on at91 sama5d2 platforms. No output is seen, unless earlyprintk is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

I've attached a log that I'm seeing on a sama5d2_xplained (sama5_defconfig and arch/arm/boot/dts/at91-sama5d2_xplained.dts). I enabled the following logs:
1. The ones in device_links_check_suppliers()
2. The ones in device_link_add()
3. initcall_debug=1

There seem to be some probe fails due to the pmc supplier not being ready:
calling at_xdmac_init+0x0/0x18 @ 1
platform f0010000.dma-controller: probe deferral - supplier f0014000.pmc not ready platform f0004000.dma-controller: probe deferral - supplier f0014000.pmc not ready initcall at_xdmac_init+0x0/0x18 returned -19 after 19531 usecs

Quoted 5 lines

calling udc_driver_init+0x0/0x18 @ 1
platform 300000.gadget: probe deferral - supplier f0014000.pmc not ready initcall udc_driver_init+0x0/0x18 returned -19 after 7524 usecs

There are others too. I'm checking them.

Thanks Tudor. I'll look into this within a few days. I'm also looking into coming up with a more generic solution.

-Saravana

#46 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Hi, Saravana,

Quoted 49 lines · Saravana Kannan

EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe

Quoted 46 lines · On Mon, Jan 25, 2021 at 9:05 AM <[email protected]> wrote:

Hi, Saravana,

Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

next-20210125 fails to boot on at91 sama5d2 platforms. No output is seen, unless earlyprintk is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

I've attached a log that I'm seeing on a sama5d2_xplained (sama5_defconfig and arch/arm/boot/dts/at91-sama5d2_xplained.dts). I enabled the following logs:
1. The ones in device_links_check_suppliers()
2. The ones in device_link_add()
3. initcall_debug=1

There seem to be some probe fails due to the pmc supplier not being ready:
calling at_xdmac_init+0x0/0x18 @ 1
platform f0010000.dma-controller: probe deferral - supplier f0014000.pmc not ready platform f0004000.dma-controller: probe deferral - supplier f0014000.pmc not ready initcall at_xdmac_init+0x0/0x18 returned -19 after 19531 usecs

Quoted 9 lines
Quoted 5 lines

calling udc_driver_init+0x0/0x18 @ 1
platform 300000.gadget: probe deferral - supplier f0014000.pmc not ready initcall udc_driver_init+0x0/0x18 returned -19 after 7524 usecs

There are others too. I'm checking them.

Thanks Tudor. I'll look into this within a few days. I'm also looking into coming up with a more generic solution.

I've sent a patch addressing this at:
https://lore.kernel.org/lkml/[email protected]/T/#u

Can you please take a look?
Cheers,
ta

#47 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 59 lines · On Thu, Jan 28, 2021 at 2:59 AM <[email protected]> wrote:

Hi, Saravana,

Quoted 55 lines · Saravana Kannan

EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe

Quoted 51 lines · On Mon, Jan 25, 2021 at 9:05 AM <[email protected]> wrote:

Hi, Saravana,

Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

next-20210125 fails to boot on at91 sama5d2 platforms. No output is seen, unless earlyprintk is enabled.

I have bisected this to commit e590474768f1cc04 ("driver core: Set fw_devlink=on by default").

I've attached a log that I'm seeing on a sama5d2_xplained (sama5_defconfig and arch/arm/boot/dts/at91-sama5d2_xplained.dts). I enabled the following logs:
1. The ones in device_links_check_suppliers()
2. The ones in device_link_add()
3. initcall_debug=1

There seem to be some probe fails due to the pmc supplier not being ready:
calling at_xdmac_init+0x0/0x18 @ 1
platform f0010000.dma-controller: probe deferral - supplier f0014000.pmc not ready platform f0004000.dma-controller: probe deferral - supplier f0014000.pmc not ready initcall at_xdmac_init+0x0/0x18 returned -19 after 19531 usecs

calling udc_driver_init+0x0/0x18 @ 1
platform 300000.gadget: probe deferral - supplier f0014000.pmc not ready initcall udc_driver_init+0x0/0x18 returned -19 after 7524 usecs

There are others too. I'm checking them.

Quoted 8 lines
Quoted 3 lines

Thanks Tudor. I'll look into this within a few days. I'm also looking into coming up with a more generic solution.

I've sent a patch addressing this at:
https://lore.kernel.org/lkml/[email protected]/T/#u

Can you please take a look?

Thanks for taking a look at this. I responded in that thread.

-Saravana

#48 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

In reply to Saravana Kannan
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

This patch breaks nios2 boot tests in qemu. The system gets stuck when trying to reboot. Reverting this patch fixes the problem. Bisect log is attached.

It may also break a variety of other boot tests, but with 115 of 430 boot tests failing in -next it is difficult to identify all culprits.

Guenter

---
Bisect log:

# bad: [a4bfd8d46ac357c12529e4eebb6c89502b03ecc9] Add linux-next specific files for 20210209
# good: [92bf22614b21a2706f4993b278017e437f7785b3] Linux 5.11-rc7
git bisect start 'HEAD' 'v5.11-rc7'
# good: [a8eb921ba7e8e77d994a1c6c69c8ef08456ecf53] Merge remote-tracking branch 'crypto/master' git bisect good a8eb921ba7e8e77d994a1c6c69c8ef08456ecf53
# good: [21d507c41bdf83f6afc0e02976e43c10badfc6cd] Merge remote-tracking branch 'spi/for-next'
git bisect good 21d507c41bdf83f6afc0e02976e43c10badfc6cd
# bad: [30cd4c688a3bcf324f011d7716044b1a4681efc1] Merge remote-tracking branch 'soundwire/next' git bisect bad 30cd4c688a3bcf324f011d7716044b1a4681efc1
# good: [c43d2173d3eb4047bb62a7a393a298a1032cce18] Merge remote-tracking branch 'drivers-x86/for-next' git bisect good c43d2173d3eb4047bb62a7a393a298a1032cce18
# bad: [4dd66c506de68a592f2dd4ef64cc9b0d7c0f3117] Merge remote-tracking branch 'usb-chipidea-next/for-usb-next' git bisect bad 4dd66c506de68a592f2dd4ef64cc9b0d7c0f3117
# good: [29b01295a829fba7399ee84afff4e64660e49f04] usb: typec: Add typec_partner_set_pd_revision git bisect good 29b01295a829fba7399ee84afff4e64660e49f04
# bad: [dac8ab120e531bf7c358b85750338e1b3d3ca0b9] Merge remote-tracking branch 'usb/usb-next'
git bisect bad dac8ab120e531bf7c358b85750338e1b3d3ca0b9
# good: [678481467d2e1460a49e626d8e9ba0c7e9742f53] usb: dwc3: core: Check maximum_speed SSP genXxY git bisect good 678481467d2e1460a49e626d8e9ba0c7e9742f53
# good: [5d3d0a61479847a8729ffdda33867f6b3443c15f] Merge remote-tracking branch 'ipmi/for-next' git bisect good 5d3d0a61479847a8729ffdda33867f6b3443c15f
# bad: [e13f5b7a130f7b6d4d34be27a87393890b5ee2ba] of: property: Add fw_devlink support for "gpio" and "gpios" binding git bisect bad e13f5b7a130f7b6d4d34be27a87393890b5ee2ba
# good: [b0e2fa4f611bb9ab22928605d5b1c7fd44e73955] driver core: Handle cycles in device links created by fw_devlink git bisect good b0e2fa4f611bb9ab22928605d5b1c7fd44e73955
# bad: [0fab972eef49ef8d30eb91d6bd98861122d083d1] drivers: core: Detach device from power domain on shutdown git bisect bad 0fab972eef49ef8d30eb91d6bd98861122d083d1
# bad: [e590474768f1cc04852190b61dec692411b22e2a] driver core: Set fw_devlink=on by default
git bisect bad e590474768f1cc04852190b61dec692411b22e2a
# good: [c13b827927112ba6170bea31c638a8573c127461] driver core: fw_devlink_relax_cycle() can be static git bisect good c13b827927112ba6170bea31c638a8573c127461

# first bad commit: [e590474768f1cc04852190b61dec692411b22e2a] driver core: Set fw_devlink=on by default
#49 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 22 lines · Guenter Roeck
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 6 lines
Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

This patch breaks nios2 boot tests in qemu. The system gets stuck when trying to reboot. Reverting this patch fixes the problem. Bisect log is attached.

Thanks for the report Guenter. Can you please try this series?
https://lore.kernel.org/lkml/[email protected]/

It's in driver-core-testing too if that's easier.

-Saravana

#50 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 26 lines · Saravana Kannan
Quoted 25 lines · Guenter Roeck
Quoted 22 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

Quoted 7 lines
Quoted 3 lines

This patch breaks nios2 boot tests in qemu. The system gets stuck when trying to reboot. Reverting this patch fixes the problem. Bisect log is attached.

Thanks for the report Guenter. Can you please try this series?
https://lore.kernel.org/lkml/[email protected]/

Not this week. I have lots of reviews to complete before the end of the week, with the 5.12 commit window coming up.

Given the number of problems observed, I personally think that it is way too early for this patch. We'll have no end of problems if it is applied to the upstream kernel in the next commit window. Of course, that is just my personal opinion.

Thanks,
Guenter

#51 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 32 lines · Guenter Roeck
Quoted 30 lines · Saravana Kannan
Quoted 28 lines · Guenter Roeck
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

This patch breaks nios2 boot tests in qemu. The system gets stuck when trying to reboot. Reverting this patch fixes the problem. Bisect log is attached.

Quoted 6 lines
Quoted 3 lines

Thanks for the report Guenter. Can you please try this series?
https://lore.kernel.org/lkml/[email protected]/

Not this week. I have lots of reviews to complete before the end of the week, with the 5.12 commit window coming up.

Ok. By next week, all the fixes should be in linux-next too. So it should be easier if you choose to test.

Quoted 4 lines

Given the number of problems observed, I personally think that it is way too early for this patch. We'll have no end of problems if it is applied to the upstream kernel in the next commit window. Of course, that is just my personal opinion.

You had said "with 115 of 430 boot tests failing in -next" earlier. Just to be sure I understand it right, you are not saying this patch caused them all right? You are just saying that 115 general boot failures that might mask fw_devlink issues in some of them, right?

Thanks,
Saravana

#52 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 48 lines · Saravana Kannan
Quoted 32 lines · Guenter Roeck
Quoted 30 lines · Saravana Kannan
Quoted 28 lines · Guenter Roeck
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

This patch breaks nios2 boot tests in qemu. The system gets stuck when trying to reboot. Reverting this patch fixes the problem. Bisect log is attached.

Quoted 6 lines
Quoted 3 lines

Thanks for the report Guenter. Can you please try this series?
https://lore.kernel.org/lkml/[email protected]/

Not this week. I have lots of reviews to complete before the end of the week, with the 5.12 commit window coming up.

Ok. By next week, all the fixes should be in linux-next too. So it should be easier if you choose to test.

Quoted 4 lines

Given the number of problems observed, I personally think that it is way too early for this patch. We'll have no end of problems if it is applied to the upstream kernel in the next commit window. Of course, that is just my personal opinion.

Quoted 5 lines

You had said "with 115 of 430 boot tests failing in -next" earlier. Just to be sure I understand it right, you are not saying this patch caused them all right? You are just saying that 115 general boot failures that might mask fw_devlink issues in some of them, right?

Correct.

Guenter

#53 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 50 lines · Guenter Roeck
Quoted 48 lines · Saravana Kannan
Quoted 32 lines · Guenter Roeck
Quoted 30 lines · Saravana Kannan
Quoted 28 lines · Guenter Roeck
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

This patch breaks nios2 boot tests in qemu. The system gets stuck when trying to reboot. Reverting this patch fixes the problem. Bisect log is attached.

Quoted 6 lines
Quoted 3 lines

Thanks for the report Guenter. Can you please try this series?
https://lore.kernel.org/lkml/[email protected]/

Not this week. I have lots of reviews to complete before the end of the week, with the 5.12 commit window coming up.

Ok. By next week, all the fixes should be in linux-next too. So it should be easier if you choose to test.

Quoted 4 lines

Given the number of problems observed, I personally think that it is way too early for this patch. We'll have no end of problems if it is applied to the upstream kernel in the next commit window. Of course, that is just my personal opinion.

Quoted 7 lines
Quoted 5 lines

You had said "with 115 of 430 boot tests failing in -next" earlier. Just to be sure I understand it right, you are not saying this patch caused them all right? You are just saying that 115 general boot failures that might mask fw_devlink issues in some of them, right?

Correct.

Is it right to assume [1] fixed all known boot issues due to fw_devlink=on?
[1] - https://lore.kernel.org/lkml/[email protected]/

-Saravana

#54 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 57 lines · Saravana Kannan
Quoted 56 lines · Guenter Roeck
Quoted 53 lines · Saravana Kannan
Quoted 32 lines · Guenter Roeck
Quoted 30 lines · Saravana Kannan
Quoted 28 lines · Guenter Roeck
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

This patch breaks nios2 boot tests in qemu. The system gets stuck when trying to reboot. Reverting this patch fixes the problem. Bisect log is attached.

Quoted 6 lines
Quoted 3 lines

Thanks for the report Guenter. Can you please try this series?
https://lore.kernel.org/lkml/[email protected]/

Not this week. I have lots of reviews to complete before the end of the week, with the 5.12 commit window coming up.

Ok. By next week, all the fixes should be in linux-next too. So it should be easier if you choose to test.

Quoted 4 lines

Given the number of problems observed, I personally think that it is way too early for this patch. We'll have no end of problems if it is applied to the upstream kernel in the next commit window. Of course, that is just my personal opinion.

You had said "with 115 of 430 boot tests failing in -next" earlier. Just to be sure I understand it right, you are not saying this patch caused them all right? You are just saying that 115 general boot failures that might mask fw_devlink issues in some of them, right?

Quoted 5 lines
Quoted 1 line

Correct.

Is it right to assume [1] fixed all known boot issues due to fw_devlink=on?
[1] - https://lore.kernel.org/lkml/[email protected]/

I honestly don't know. Current status of -next in my tests is:

Build results:

	total: 149 pass: 144 fail: 5
Qemu test results:
	total: 432 pass: 371 fail: 61

This is for next-20210216. Newly introduced failures keep popping up. Some of the failures have been persistent for weeks, so it is all but impossible to say if affected platforms experience more than one failure.

Also, please keep in mind that my boot tests are very shallow, along the line of "it boots, therefore it works". It only tests hardware which is emulated by qemu and is needed for booting. It tests probably much less than 1% of driver code. It can and should not be used for any useful fw_devlink related test coverage.

Thanks,
Guenter

#55 lore raw

Re: [PATCH v1 5/5] driver core: Set fw_devlink=on by default

Quoted 76 lines · Guenter Roeck
Quoted 57 lines · Saravana Kannan
Quoted 56 lines · Guenter Roeck
Quoted 53 lines · Saravana Kannan
Quoted 32 lines · Guenter Roeck
Quoted 30 lines · Saravana Kannan
Quoted 28 lines · Guenter Roeck
Quoted 20 lines · Saravana Kannan

Cyclic dependencies in some firmware was one of the last remaining reasons fw_devlink=on couldn't be set by default. Now that cyclic dependencies don't block probing, set fw_devlink=on by default.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

If this patch prevents some devices from probing, it's very likely due to the system having one or more device drivers that "probe"/set up a device (DT node with compatible property) without creating a struct device for it. If we hit such cases, the device drivers need to be fixed so that they populate struct devices and probe them like normal device drivers so that the driver core is aware of the devices and their status. See [1] for an example of such a case.

Quoted 2 lines

[1] - https://lore.kernel.org/lkml/CAGETcx9PiX==mLxB9PO8Myyk6u2vhPVwTMsA5NkD-ywH5xhusw@mail.gmail.com/

This patch breaks nios2 boot tests in qemu. The system gets stuck when trying to reboot. Reverting this patch fixes the problem. Bisect log is attached.

Quoted 6 lines
Quoted 3 lines

Thanks for the report Guenter. Can you please try this series?
https://lore.kernel.org/lkml/[email protected]/

Not this week. I have lots of reviews to complete before the end of the week, with the 5.12 commit window coming up.

Ok. By next week, all the fixes should be in linux-next too. So it should be easier if you choose to test.

Quoted 4 lines

Given the number of problems observed, I personally think that it is way too early for this patch. We'll have no end of problems if it is applied to the upstream kernel in the next commit window. Of course, that is just my personal opinion.

You had said "with 115 of 430 boot tests failing in -next" earlier. Just to be sure I understand it right, you are not saying this patch caused them all right? You are just saying that 115 general boot failures that might mask fw_devlink issues in some of them, right?

Quoted 5 lines
Quoted 1 line

Correct.

Is it right to assume [1] fixed all known boot issues due to fw_devlink=on?
[1] - https://lore.kernel.org/lkml/[email protected]/

I honestly don't know. Current status of -next in my tests is:

Build results:

        total: 149 pass: 144 fail: 5
Qemu test results:
        total: 432 pass: 371 fail: 61

This is for next-20210216. Newly introduced failures keep popping up. Some of the failures have been persistent for weeks, so it is all but impossible to say if affected platforms experience more than one failure.

Quoted 5 lines

Also, please keep in mind that my boot tests are very shallow, along the line of "it boots, therefore it works". It only tests hardware which is emulated by qemu and is needed for booting. It tests probably much less than 1% of driver code. It can and should not be used for any useful fw_devlink related test coverage.

Agreed. I'm not using this for fw_devlink=on test coverage. Just checking to make sure I've addressed any issues you've seen.

FYI, you can change it at runtime using the kernel commandline param fw_devlink=permissive. So, you don't have to build all these kernels again to test if fw_devlink=on is making things worse.

-Saravana

#56 lore raw
In reply to Saravana Kannan
Quoted 26 lines · Saravana Kannan

As discussed in LPC 2020, cyclic dependencies in firmware that couldn't be broken using logic was one of the last remaining reasons fw_devlink=on couldn't be set by default.

This series changes fw_devlink so that when a cyclic dependency is found in firmware, the links between those devices fallback to permissive mode behavior. This way, the rest of the system still benefits from fw_devlink, but the ambiguous cases fallback to permissive mode.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

Greg/Rafael,

Can we get this pulled into 5.11-rc1 or -rc2 soon please? I expect to see some issues due to device drivers that aren't following best practices (they don't expose the device to driver core). Want to identify those early on and try to have them fixed before 5.11 release. See [1] for an example of such a case.

Quoted 7 lines

If we do end up have to revert anything, it'll just be Patch 5/5 (a one liner).

Marc,

You had hit issues with fw_devlink=on before on some of your systems. Want to give this a shot?

Marc,

If you decide to test this, please also pull in this patch. It should fix all your interrupt issues.

https://lore.kernel.org/lkml/[email protected]/

-Saravana

#57 lore raw
In reply to Saravana Kannan
Quoted 30 lines · On Thu, 17 Dec 2020 19:16:58 -0800 Saravana Kannan wrote:

As discussed in LPC 2020, cyclic dependencies in firmware that couldn't be broken using logic was one of the last remaining reasons fw_devlink=on couldn't be set by default.

This series changes fw_devlink so that when a cyclic dependency is found in firmware, the links between those devices fallback to permissive mode behavior. This way, the rest of the system still benefits from fw_devlink, but the ambiguous cases fallback to permissive mode.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

Greg/Rafael,

Can we get this pulled into 5.11-rc1 or -rc2 soon please? I expect to see some issues due to device drivers that aren't following best practices (they don't expose the device to driver core). Want to identify those early on and try to have them fixed before 5.11 release. See [1] for an example of such a case.

If we do end up have to revert anything, it'll just be Patch 5/5 (a one liner).

Quoted 9 lines

Marc,

You had hit issues with fw_devlink=on before on some of your systems. Want to give this a shot?

Jisheng,

Want to fix up one of those gpio drivers you were having problems with?

Hi Saravana,

I didn't send fix for the gpio-dwapb.c in last development window, so can send patch once 5.11-rc1 is released.

thanks

A message that is not in the archive (sent off-list or not yet received): CAHp75VfqL1QuvjCZ7p23e_2qhY3DUgVNaS--Uk1mEoEHsD8GBA@mail.gmail.com

#59 lore raw
Quoted 48 lines · Andy Shevchenko
Quoted 44 lines · Jisheng Zhang
Quoted 39 lines · On Thu, 17 Dec 2020 19:16:58 -0800 Saravana Kannan wrote:

As discussed in LPC 2020, cyclic dependencies in firmware that couldn't be broken using logic was one of the last remaining reasons fw_devlink=on couldn't be set by default.

This series changes fw_devlink so that when a cyclic dependency is found in firmware, the links between those devices fallback to permissive mode behavior. This way, the rest of the system still benefits from fw_devlink, but the ambiguous cases fallback to permissive mode.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

Greg/Rafael,

Can we get this pulled into 5.11-rc1 or -rc2 soon please? I expect to see some issues due to device drivers that aren't following best practices (they don't expose the device to driver core). Want to identify those early on and try to have them fixed before 5.11 release. See [1] for an example of such a case.

If we do end up have to revert anything, it'll just be Patch 5/5 (a one liner).

Marc,

You had hit issues with fw_devlink=on before on some of your systems. Want to give this a shot?

Jisheng,

Want to fix up one of those gpio drivers you were having problems with?

Quoted 7 lines
Quoted 4 lines

Hi Saravana,

I didn't send fix for the gpio-dwapb.c in last development window, so can send patch once 5.11-rc1 is released.

If you are going to do anything with that GPIO driver, it should be removal of compatible strings from the device child nodes. The driver IIRC never used them anyhow anyway.

We already discussed this in a different thread. Just deleting DT is not okay. That breaks a new kernel + old DT combo. Upgrading the kernel shouldn't break a board.

-Saravana

#60 lore raw
In reply to Saravana Kannan
Quoted 18 lines · Saravana Kannan

As discussed in LPC 2020, cyclic dependencies in firmware that couldn't be broken using logic was one of the last remaining reasons fw_devlink=on couldn't be set by default.

This series changes fw_devlink so that when a cyclic dependency is found in firmware, the links between those devices fallback to permissive mode behavior. This way, the rest of the system still benefits from fw_devlink, but the ambiguous cases fallback to permissive mode.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

Quoted 3 lines

Greg/Rafael,

Can we get this pulled into 5.11-rc1 or -rc2 soon please?

Honestly, I'd rather not (but it's up to Greg).

This is a new series posted during the merge window, so it should not be looked at even according to the rules.

Personally, I don't have the time to look at it now.

Quoted 4 lines

I expect to see some issues due to device drivers that aren't following best practices (they don't expose the device to driver core). Want to identify those early on and try to have them fixed before 5.11 release. See [1] for an example of such a case.

So it should be posted right after -rc1 and spend a whole cycle in linux-next.

Quoted 2 lines

If we do end up have to revert anything, it'll just be Patch 5/5 (a one liner).

Which totally doesn't matter IMV.

#61 lore raw
In reply to Saravana Kannan
Quoted 17 lines · Saravana Kannan

As discussed in LPC 2020, cyclic dependencies in firmware that couldn't be broken using logic was one of the last remaining reasons fw_devlink=on couldn't be set by default.

This series changes fw_devlink so that when a cyclic dependency is found in firmware, the links between those devices fallback to permissive mode behavior. This way, the rest of the system still benefits from fw_devlink, but the ambiguous cases fallback to permissive mode.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

Quoted 7 lines

Greg/Rafael,

Can we get this pulled into 5.11-rc1 or -rc2 soon please? I expect to see some issues due to device drivers that aren't following best practices (they don't expose the device to driver core). Want to identify those early on and try to have them fixed before 5.11 release. See [1] for an example of such a case.

Now queued up in my tree, will show up in linux-next in a few days, let's see what breaks! :)

And it is scheduled for 5.12-rc1, not 5.11, sorry.

thanks,

greg k-h

#62 lore raw
Quoted 27 lines · Greg Kroah-Hartman
Quoted 17 lines · Saravana Kannan

As discussed in LPC 2020, cyclic dependencies in firmware that couldn't be broken using logic was one of the last remaining reasons fw_devlink=on couldn't be set by default.

This series changes fw_devlink so that when a cyclic dependency is found in firmware, the links between those devices fallback to permissive mode behavior. This way, the rest of the system still benefits from fw_devlink, but the ambiguous cases fallback to permissive mode.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

Quoted 7 lines

Greg/Rafael,

Can we get this pulled into 5.11-rc1 or -rc2 soon please? I expect to see some issues due to device drivers that aren't following best practices (they don't expose the device to driver core). Want to identify those early on and try to have them fixed before 5.11 release. See [1] for an example of such a case.

Quoted 4 lines

Now queued up in my tree, will show up in linux-next in a few days, let's see what breaks! :)

And it is scheduled for 5.12-rc1, not 5.11, sorry.

Thanks. Not too worried about the actual version. I just want things to start breaking as soon as possible if they are going to break.

-Saravana

#63 lore raw
In reply to Greg Kroah-Hartman
Quoted 26 lines · Greg Kroah-Hartman
Quoted 17 lines · Saravana Kannan

As discussed in LPC 2020, cyclic dependencies in firmware that couldn't be broken using logic was one of the last remaining reasons fw_devlink=on couldn't be set by default.

This series changes fw_devlink so that when a cyclic dependency is found in firmware, the links between those devices fallback to permissive mode behavior. This way, the rest of the system still benefits from fw_devlink, but the ambiguous cases fallback to permissive mode.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

Quoted 7 lines

Greg/Rafael,

Can we get this pulled into 5.11-rc1 or -rc2 soon please? I expect to see some issues due to device drivers that aren't following best practices (they don't expose the device to driver core). Want to identify those early on and try to have them fixed before 5.11 release. See [1] for an example of such a case.

Quoted 4 lines

Now queued up in my tree, will show up in linux-next in a few days, let's see what breaks! :)

And it is scheduled for 5.12-rc1, not 5.11, sorry.

For the record, this breaks my rk3399 board, (NanoPC-T4) as no mass storage can be discovered (it lives on PCIe):

(initramfs) find /sys -name 'waiting_for_supplier'| xargs grep .| egrep -v ':0$' /sys/devices/platform/ff3d0000.i2c/i2c-4/4-0022/waiting_for_supplier:1 /sys/devices/platform/f8000000.pcie/waiting_for_supplier:1 /sys/devices/platform/fe320000.mmc/waiting_for_supplier:1 /sys/devices/platform/sdio-pwrseq/waiting_for_supplier:1 /sys/devices/platform/ff3c0000.i2c/i2c-0/0-001b/waiting_for_supplier:1

Enabling the debug prints in device_links_check_suppliers(), I end up with the dump below (apologies for the size).

This seems to all hang on the GPIO banks, but it is pretty unclear what is wrong with them.

Happy to test things further.

        M.

platform vcc3v3-sys: probe deferral - supplier vcc12v0-sys not ready platform vcc5v0-sys: probe deferral - supplier vcc12v0-sys not ready platform vcc1v8-s3: probe deferral - wait for supplier pmic@1b platform vcc3v0-sd: probe deferral - supplier vcc3v3-sys not ready platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform vbus-typec: probe deferral - supplier vcc5v0-sys not ready platform vcc5v0-host0: probe deferral - supplier vcc5v0-sys not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - wait for supplier pmic@1b platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - wait for supplier pmic@1b platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - wait for supplier pmic@1b platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - wait for supplier pmic@1b platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - wait for supplier pmic@1b platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - wait for supplier pmic@1b platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform fe320000.mmc: probe deferral - wait for supplier pmic@1b platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - wait for supplier pmic@1b platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform fe320000.mmc: probe deferral - wait for supplier pmic@1b platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - supplier 0-001b not ready platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
platform fe320000.mmc: probe deferral - wait for supplier
gpio0@ff720000
platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - supplier 0-001b not ready platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
platform fe320000.mmc: probe deferral - wait for supplier
gpio0@ff720000
platform fe300000.ethernet: probe deferral - supplier 0-001b not ready platform fe380000.usb: probe deferral - supplier
ff770000.syscon:usb2-phy@e450 not ready
platform fe3c0000.usb: probe deferral - supplier
ff770000.syscon:usb2-phy@e460 not ready
platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - supplier 0-001b not ready platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
platform fe3a0000.usb: probe deferral - supplier
ff770000.syscon:usb2-phy@e450 not ready
platform fe3e0000.usb: probe deferral - supplier
ff770000.syscon:usb2-phy@e460 not ready
platform fe320000.mmc: probe deferral - wait for supplier
gpio0@ff720000
platform fe300000.ethernet: probe deferral - supplier 0-001b not ready platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - supplier 0-001b not ready platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
i2c 0-001b: probe deferral - wait for supplier gpio1@ff730000 platform fe320000.mmc: probe deferral - wait for supplier
gpio0@ff720000
platform fe300000.ethernet: probe deferral - supplier 0-001b not ready platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - supplier 0-001b not ready platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
i2c 0-001b: probe deferral - wait for supplier gpio1@ff730000 platform fe320000.mmc: probe deferral - wait for supplier
gpio0@ff720000
platform fe300000.ethernet: probe deferral - supplier 0-001b not ready platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - supplier 0-001b not ready platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
i2c 0-001b: probe deferral - wait for supplier gpio1@ff730000 platform fe320000.mmc: probe deferral - wait for supplier
gpio0@ff720000
platform fe300000.ethernet: probe deferral - supplier 0-001b not ready platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - supplier 0-001b not ready platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
i2c 0-001b: probe deferral - wait for supplier gpio1@ff730000 platform fe320000.mmc: probe deferral - wait for supplier
gpio0@ff720000
platform fe300000.ethernet: probe deferral - supplier 0-001b not ready platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000 platform vcc1v8-s3: probe deferral - supplier 0-001b not ready platform vcca0v9-s3: probe deferral - supplier vcc1v8-s3 not ready platform vcca1v8-s3: probe deferral - supplier vcc1v8-s3 not ready platform f8000000.pcie: probe deferral - wait for supplier
gpio2@ff780000
i2c 0-001b: probe deferral - wait for supplier gpio1@ff730000 platform fe320000.mmc: probe deferral - wait for supplier
gpio0@ff720000
platform fe300000.ethernet: probe deferral - supplier 0-001b not ready platform sdio-pwrseq: probe deferral - wait for supplier gpio0@ff720000

Signature · 1 line
Jazz is not dead. It just smells funny...
maintainer
#64 lore raw
Quoted 43 lines · Marc Zyngier
Quoted 26 lines · Greg Kroah-Hartman
Quoted 17 lines · Saravana Kannan

As discussed in LPC 2020, cyclic dependencies in firmware that couldn't be broken using logic was one of the last remaining reasons fw_devlink=on couldn't be set by default.

This series changes fw_devlink so that when a cyclic dependency is found in firmware, the links between those devices fallback to permissive mode behavior. This way, the rest of the system still benefits from fw_devlink, but the ambiguous cases fallback to permissive mode.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

Quoted 7 lines

Greg/Rafael,

Can we get this pulled into 5.11-rc1 or -rc2 soon please? I expect to see some issues due to device drivers that aren't following best practices (they don't expose the device to driver core). Want to identify those early on and try to have them fixed before 5.11 release. See [1] for an example of such a case.

Quoted 4 lines

Now queued up in my tree, will show up in linux-next in a few days, let's see what breaks!  :)

And it is scheduled for 5.12-rc1, not 5.11, sorry.

For the record, this breaks my rk3399 board, (NanoPC-T4) as no mass storage can be discovered (it lives on PCIe):

(initramfs) find /sys -name 'waiting_for_supplier'| xargs grep .| egrep -v ':0$' /sys/devices/platform/ff3d0000.i2c/i2c-4/4-0022/waiting_for_supplier:1 /sys/devices/platform/f8000000.pcie/waiting_for_supplier:1 /sys/devices/platform/fe320000.mmc/waiting_for_supplier:1 /sys/devices/platform/sdio-pwrseq/waiting_for_supplier:1 /sys/devices/platform/ff3c0000.i2c/i2c-0/0-001b/waiting_for_supplier:1

Quoted 2 lines

Enabling the debug prints in device_links_check_suppliers(), I end up with the dump below (apologies for the size).

I am seeing the same problem on Tegra30 Cardhu A04 where several regulators are continuously deferred and prevents the board from booting ...

[    2.518334] platform panel: probe deferral - supplier regulator@11 not ready
[    2.525503] platform regulator@1: probe deferral - supplier 4-002d not ready
[    2.533141] platform regulator@3: probe deferral - supplier regulator@101 not ready
[    2.540856] platform regulator@5: probe deferral - supplier regulator@101 not ready
[    2.548589] platform regulator@6: probe deferral - supplier regulator@101 not ready
[    2.556316] platform regulator@7: probe deferral - supplier regulator@101 not ready
[    2.564041] platform regulator@8: probe deferral - supplier regulator@101 not ready
[    2.571743] platform regulator@9: probe deferral - supplier regulator@101 not ready
[    2.579463] platform regulator@10: probe deferral - supplier regulator@101 not ready
[    2.587273] platform regulator@11: probe deferral - supplier regulator@101 not ready
[    2.595088] platform regulator@12: probe deferral - supplier regulator@104 not ready
[    2.603837] platform regulator@102: probe deferral - supplier regulator@104 not ready
[    2.611726] platform regulator@103: probe deferral - supplier regulator@104 not ready
[    2.620137] platform 3000.pcie: probe deferral - supplier regulator@5 not ready

Cheers
Jon

Signature · 1 line
nvpublic
#65 lore raw
Quoted 73 lines · Jon Hunter
Quoted 43 lines · Marc Zyngier
Quoted 26 lines · Greg Kroah-Hartman
Quoted 17 lines · Saravana Kannan

As discussed in LPC 2020, cyclic dependencies in firmware that couldn't be broken using logic was one of the last remaining reasons fw_devlink=on couldn't be set by default.

This series changes fw_devlink so that when a cyclic dependency is found in firmware, the links between those devices fallback to permissive mode behavior. This way, the rest of the system still benefits from fw_devlink, but the ambiguous cases fallback to permissive mode.

Setting fw_devlink=on by default brings a bunch of benefits (currently, only for systems with device tree firmware):
* Significantly cuts down deferred probes.
* Device probe is effectively attempted in graph order.
* Makes it much easier to load drivers as modules without having to worry about functional dependencies between modules (depmod is still needed for symbol dependencies).

Quoted 7 lines

Greg/Rafael,

Can we get this pulled into 5.11-rc1 or -rc2 soon please? I expect to see some issues due to device drivers that aren't following best practices (they don't expose the device to driver core). Want to identify those early on and try to have them fixed before 5.11 release. See [1] for an example of such a case.

Quoted 4 lines

Now queued up in my tree, will show up in linux-next in a few days, let's see what breaks! :)

And it is scheduled for 5.12-rc1, not 5.11, sorry.

For the record, this breaks my rk3399 board, (NanoPC-T4) as no mass storage can be discovered (it lives on PCIe):

(initramfs) find /sys -name 'waiting_for_supplier'| xargs grep .| egrep -v ':0$' /sys/devices/platform/ff3d0000.i2c/i2c-4/4-0022/waiting_for_supplier:1 /sys/devices/platform/f8000000.pcie/waiting_for_supplier:1 /sys/devices/platform/fe320000.mmc/waiting_for_supplier:1 /sys/devices/platform/sdio-pwrseq/waiting_for_supplier:1 /sys/devices/platform/ff3c0000.i2c/i2c-0/0-001b/waiting_for_supplier:1

Quoted 2 lines

Enabling the debug prints in device_links_check_suppliers(), I end up with the dump below (apologies for the size).

I am seeing the same problem on Tegra30 Cardhu A04 where several regulators are continuously deferred and prevents the board from booting ...

[    2.518334] platform panel: probe deferral - supplier regulator@11 not ready
[    2.525503] platform regulator@1: probe deferral - supplier 4-002d not ready
[    2.533141] platform regulator@3: probe deferral - supplier regulator@101 not ready
[    2.540856] platform regulator@5: probe deferral - supplier regulator@101 not ready
[    2.548589] platform regulator@6: probe deferral - supplier regulator@101 not ready
[    2.556316] platform regulator@7: probe deferral - supplier regulator@101 not ready
[    2.564041] platform regulator@8: probe deferral - supplier regulator@101 not ready
[    2.571743] platform regulator@9: probe deferral - supplier regulator@101 not ready
[    2.579463] platform regulator@10: probe deferral - supplier regulator@101 not ready
[    2.587273] platform regulator@11: probe deferral - supplier regulator@101 not ready
Quoted 7 lines
[    2.595088] platform regulator@12: probe deferral - supplier regulator@104 not ready
[    2.603837] platform regulator@102: probe deferral - supplier regulator@104 not ready
[    2.611726] platform regulator@103: probe deferral - supplier regulator@104 not ready
[    2.620137] platform 3000.pcie: probe deferral - supplier regulator@5 not ready

Looks like this is not the whole log? Do you see any "wait for supplier" logs? That's what all these boot issues should boil down to. And as usual, pointer to DT for this board please.

-Saravana

maintainer
#66 lore raw

On 13/01/2021 21:29, Saravana Kannan wrote:

...

Quoted 27 lines
Quoted 27 lines

I am seeing the same problem on Tegra30 Cardhu A04 where several regulators are continuously deferred and prevents the board from booting ...

[    2.518334] platform panel: probe deferral - supplier regulator@11 not ready
[    2.525503] platform regulator@1: probe deferral - supplier 4-002d not ready
[    2.533141] platform regulator@3: probe deferral - supplier regulator@101 not ready
[    2.540856] platform regulator@5: probe deferral - supplier regulator@101 not ready
[    2.548589] platform regulator@6: probe deferral - supplier regulator@101 not ready
[    2.556316] platform regulator@7: probe deferral - supplier regulator@101 not ready
[    2.564041] platform regulator@8: probe deferral - supplier regulator@101 not ready
[    2.571743] platform regulator@9: probe deferral - supplier regulator@101 not ready
[    2.579463] platform regulator@10: probe deferral - supplier regulator@101 not ready
[    2.587273] platform regulator@11: probe deferral - supplier regulator@101 not ready
[    2.595088] platform regulator@12: probe deferral - supplier regulator@104 not ready
[    2.603837] platform regulator@102: probe deferral - supplier regulator@104 not ready
Quoted 7 lines
Quoted 3 lines
[    2.611726] platform regulator@103: probe deferral - supplier regulator@104 not ready
[    2.620137] platform 3000.pcie: probe deferral - supplier regulator@5 not ready

Looks like this is not the whole log? Do you see any "wait for supplier" logs? That's what all these boot issues should boil down to. And as usual, pointer to DT for this board please.

Ah yes I see ...

platform regulator@1: probe deferral - wait for supplier tps65911@2d

Yes the device-tree for this board can be found here [0]. Looks like there is a circular dependency between the vddctrl_reg and vddcore_reg. This is part of coupled regulators which have a two-way linkage [1]. So this change appears to conflict with this.

Jon

[0]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm/boot/dts/tegra30-cardhu-a04.dts
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/regulator/regulator.yaml#n129

Signature · 1 line
nvpublic
#67 lore raw
Quoted 37 lines · Jon Hunter

On 13/01/2021 21:29, Saravana Kannan wrote:

...

Quoted 31 lines
Quoted 30 lines

I am seeing the same problem on Tegra30 Cardhu A04 where several regulators are continuously deferred and prevents the board from booting ...

[    2.518334] platform panel: probe deferral - supplier regulator@11 not ready
[    2.525503] platform regulator@1: probe deferral - supplier 4-002d not ready
[    2.533141] platform regulator@3: probe deferral - supplier regulator@101 not ready
[    2.540856] platform regulator@5: probe deferral - supplier regulator@101 not ready
[    2.548589] platform regulator@6: probe deferral - supplier regulator@101 not ready
[    2.556316] platform regulator@7: probe deferral - supplier regulator@101 not ready
[    2.564041] platform regulator@8: probe deferral - supplier regulator@101 not ready
[    2.571743] platform regulator@9: probe deferral - supplier regulator@101 not ready
[    2.579463] platform regulator@10: probe deferral - supplier regulator@101 not ready
[    2.587273] platform regulator@11: probe deferral - supplier regulator@101 not ready
[    2.595088] platform regulator@12: probe deferral - supplier regulator@104 not ready
[    2.603837] platform regulator@102: probe deferral - supplier regulator@104 not ready
[    2.611726] platform regulator@103: probe deferral - supplier regulator@104 not ready
[    2.620137] platform 3000.pcie: probe deferral - supplier regulator@5 not ready
Quoted 7 lines
Quoted 3 lines

Looks like this is not the whole log? Do you see any "wait for supplier" logs? That's what all these boot issues should boil down to. And as usual, pointer to DT for this board please.

Ah yes I see ...

platform regulator@1: probe deferral - wait for supplier tps65911@2d

Do you mind sharing the full log please? It's hard to tell you anything useful with bits and pieces of logs.

Quoted 4 lines

Yes the device-tree for this board can be found here [0]. Looks like there is a circular dependency between the vddctrl_reg and vddcore_reg. This is part of coupled regulators which have a two-way linkage [1]. So this change appears to conflict with this.

fw_devlink doesn't track "regulator-coupled-with". So that's probably not it. Also, this patch series was made to handle simple cycles properly. It'll functionally disable the device links it created when it comes to probe ordering. Only two overlapping cycles might cause issues -- and even that, not all the time. So yeah, full log please.

-Saravana

Quoted 9 lines

Jon

[0]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm/boot/dts/tegra30-cardhu-a04.dts
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/bindings/regulator/regulator.yaml#n129

--
nvpublic

maintainer
#68 lore raw
Quoted 54 lines · Saravana Kannan
Quoted 37 lines · Jon Hunter

On 13/01/2021 21:29, Saravana Kannan wrote:

...

Quoted 31 lines
Quoted 30 lines

I am seeing the same problem on Tegra30 Cardhu A04 where several regulators are continuously deferred and prevents the board from booting ...

[    2.518334] platform panel: probe deferral - supplier regulator@11 not ready
[    2.525503] platform regulator@1: probe deferral - supplier 4-002d not ready
[    2.533141] platform regulator@3: probe deferral - supplier regulator@101 not ready
[    2.540856] platform regulator@5: probe deferral - supplier regulator@101 not ready
[    2.548589] platform regulator@6: probe deferral - supplier regulator@101 not ready
[    2.556316] platform regulator@7: probe deferral - supplier regulator@101 not ready
[    2.564041] platform regulator@8: probe deferral - supplier regulator@101 not ready
[    2.571743] platform regulator@9: probe deferral - supplier regulator@101 not ready
[    2.579463] platform regulator@10: probe deferral - supplier regulator@101 not ready
[    2.587273] platform regulator@11: probe deferral - supplier regulator@101 not ready
[    2.595088] platform regulator@12: probe deferral - supplier regulator@104 not ready
[    2.603837] platform regulator@102: probe deferral - supplier regulator@104 not ready
[    2.611726] platform regulator@103: probe deferral - supplier regulator@104 not ready
[    2.620137] platform 3000.pcie: probe deferral - supplier regulator@5 not ready
Quoted 7 lines
Quoted 3 lines

Looks like this is not the whole log? Do you see any "wait for supplier" logs? That's what all these boot issues should boil down to. And as usual, pointer to DT for this board please.

Ah yes I see ...

platform regulator@1: probe deferral - wait for supplier tps65911@2d

Do you mind sharing the full log please? It's hard to tell you anything useful with bits and pieces of logs.

Quoted 4 lines

Yes the device-tree for this board can be found here [0]. Looks like there is a circular dependency between the vddctrl_reg and vddcore_reg. This is part of coupled regulators which have a two-way linkage [1]. So this change appears to conflict with this.

Quoted 5 lines

fw_devlink doesn't track "regulator-coupled-with". So that's probably not it. Also, this patch series was made to handle simple cycles properly. It'll functionally disable the device links it created when it comes to probe ordering. Only two overlapping cycles might cause issues -- and even that, not all the time. So yeah, full log please.

No problem. Please find attached.

Cheers
Jon

Signature · 1 line
nvpublic
[  111.269288] VFS: Unable to mount root fs via NFS.
[  111.274136] devtmpfs: mounted
[  111.278710] Freeing unused kernel memory: 1024K
[  111.297234] Run /sbin/init as init process
[  111.301355]   with arguments:
[  111.304328]     /sbin/init
[  111.307076]     netdevwait
[  111.309794]   with environment:
[  111.312940]     HOME=/
[  111.315303]     TERM=linux
[  111.318333] Run /etc/init as init process
[  111.322358]   with arguments:
[  111.325330]     /etc/init
[  111.327984]     netdevwait
[  111.330702]   with environment:
[  111.333847]     HOME=/
[  111.336265]     TERM=linux
[  111.339174] Run /bin/init as init process
[  111.343196]   with arguments:
[  111.346206]     /bin/init
[  111.348837]     netdevwait
[  111.351548]   with environment:
[  111.354692]     HOME=/
[  111.357110]     TERM=linux
[  111.360010] Run /bin/sh as init process
[  111.363858]   with arguments:
[  111.366860]     /bin/sh
[  111.369316]     netdevwait
[  111.372027]   with environment:
[  111.375170]     HOME=/
[  111.377608]     TERM=linux
[  111.380510] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[  111.394699] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  111.403415] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  111.409697] [<c0111410>] (unwind_backtrace) from [<c010bbb0>] (show_stack+0x10/0x14)
[  111.417489] [<c010bbb0>] (show_stack) from [<c0be51ec>] (dump_stack+0xc8/0xdc)
[  111.424750] [<c0be51ec>] (dump_stack) from [<c0be3a2c>] (panic+0x114/0x32c)
[  111.431736] [<c0be3a2c>] (panic) from [<c0bea960>] (kernel_init+0x108/0x118)
[  111.438807] [<c0bea960>] (kernel_init) from [<c01001b0>] (ret_from_fork+0x14/0x24)
[  111.446399] Exception stack(0xc1507fb0 to 0xc1507ff8)
[  111.451461] 7fa0:                                     00000000 00000000 00000000 00000000
[  111.459652] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  111.467841] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  111.474475] CPU3: stopping
[  111.477198] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  111.485914] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  111.492190] [<c0111410>] (unwind_backtrace) from [<c010bbb0>] (show_stack+0x10/0x14)
[  111.499964] [<c010bbb0>] (show_stack) from [<c0be51ec>] (dump_stack+0xc8/0xdc)
[  111.507213] [<c0be51ec>] (dump_stack) from [<c010ee98>] (do_handle_IPI+0x3ac/0x3bc)
[  111.514897] [<c010ee98>] (do_handle_IPI) from [<c010eec0>] (ipi_handler+0x18/0x20)
[  111.522490] [<c010eec0>] (ipi_handler) from [<c018ff28>] (handle_percpu_devid_irq+0x8c/0x294)
[  111.531046] [<c018ff28>] (handle_percpu_devid_irq) from [<c01899e4>] (generic_handle_irq+0x34/0x44)
[  111.540128] [<c01899e4>] (generic_handle_irq) from [<c018a088>] (__handle_domain_irq+0x5c/0xb4)
[  111.548856] [<c018a088>] (__handle_domain_irq) from [<c04feae8>] (gic_handle_irq+0x80/0x94)
[  111.557245] [<c04feae8>] (gic_handle_irq) from [<c0100b8c>] (__irq_svc+0x6c/0xa8)
[  111.564752] Exception stack(0xc1527ee0 to 0xc1527f28)
[  111.569817] 7ee0: 00000000 c120e830 2e762000 ef7cf780 00000000 c120e830 00000000 00000000
[  111.578010] 7f00: ef7ce938 c1254180 f4656888 00000019 fffffff6 c1527f30 c0838164 c0838234
[  111.586196] 7f20: 60000113 ffffffff
[  111.589690] [<c0100b8c>] (__irq_svc) from [<c0838234>] (cpuidle_enter_state+0x1c4/0x524)
[  111.597813] [<c0838234>] (cpuidle_enter_state) from [<c083a9cc>] (cpuidle_enter_state_coupled+0x144/0x3c8)
[  111.607495] [<c083a9cc>] (cpuidle_enter_state_coupled) from [<c08385f8>] (cpuidle_enter+0x50/0x54)
[  111.616477] [<c08385f8>] (cpuidle_enter) from [<c015aad0>] (do_idle+0x204/0x280)
[  111.623903] [<c015aad0>] (do_idle) from [<c015aea4>] (cpu_startup_entry+0x18/0x1c)
[  111.631495] [<c015aea4>] (cpu_startup_entry) from [<80101770>] (0x80101770)
[  111.638476] CPU1: stopping
[  111.641201] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  111.649917] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  111.656191] [<c0111410>] (unwind_backtrace) from [<c010bbb0>] (show_stack+0x10/0x14)
[  111.663964] [<c010bbb0>] (show_stack) from [<c0be51ec>] (dump_stack+0xc8/0xdc)
[  111.671214] [<c0be51ec>] (dump_stack) from [<c010ee98>] (do_handle_IPI+0x3ac/0x3bc)
[  111.678897] [<c010ee98>] (do_handle_IPI) from [<c010eec0>] (ipi_handler+0x18/0x20)
[  111.686490] [<c010eec0>] (ipi_handler) from [<c018ff28>] (handle_percpu_devid_irq+0x8c/0x294)
[  111.695042] [<c018ff28>] (handle_percpu_devid_irq) from [<c01899e4>] (generic_handle_irq+0x34/0x44)
[  111.704115] [<c01899e4>] (generic_handle_irq) from [<c018a088>] (__handle_domain_irq+0x5c/0xb4)
[  111.712842] [<c018a088>] (__handle_domain_irq) from [<c04feae8>] (gic_handle_irq+0x80/0x94)
[  111.721222] [<c04feae8>] (gic_handle_irq) from [<c0100b8c>] (__irq_svc+0x6c/0xa8)
[  111.728728] Exception stack(0xc1523ee0 to 0xc1523f28)
[  111.733794] 3ee0: 00000000 c120e830 2e73a000 ef7a7780 00000000 c120e830 00000000 00000000
[  111.741986] 3f00: ef7a6938 c1254180 f4656888 00000019 fffffff6 c1523f30 c0838164 c0838234
[  111.750171] 3f20: 60000113 ffffffff
[  111.753665] [<c0100b8c>] (__irq_svc) from [<c0838234>] (cpuidle_enter_state+0x1c4/0x524)
[  111.761780] [<c0838234>] (cpuidle_enter_state) from [<c083a9cc>] (cpuidle_enter_state_coupled+0x144/0x3c8)
[  111.771460] [<c083a9cc>] (cpuidle_enter_state_coupled) from [<c08385f8>] (cpuidle_enter+0x50/0x54)
[  111.780443] [<c08385f8>] (cpuidle_enter) from [<c015aad0>] (do_idle+0x204/0x280)
[  111.787864] [<c015aad0>] (do_idle) from [<c015aea4>] (cpu_startup_entry+0x18/0x1c)
[  111.795456] [<c015aea4>] (cpu_startup_entry) from [<80101770>] (0x80101770)
[  111.802435] CPU0: stopping
[  111.805160] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  111.813874] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  111.820147] [<c0111410>] (unwind_backtrace) from [<c010bbb0>] (show_stack+0x10/0x14)
[  111.827923] [<c010bbb0>] (show_stack) from [<c0be51ec>] (dump_stack+0xc8/0xdc)
[  111.835172] [<c0be51ec>] (dump_stack) from [<c010ee98>] (do_handle_IPI+0x3ac/0x3bc)
[  111.842853] [<c010ee98>] (do_handle_IPI) from [<c010eec0>] (ipi_handler+0x18/0x20)
[  111.850446] [<c010eec0>] (ipi_handler) from [<c018ff28>] (handle_percpu_devid_irq+0x8c/0x294)
[  111.858998] [<c018ff28>] (handle_percpu_devid_irq) from [<c01899e4>] (generic_handle_irq+0x34/0x44)
[  111.868073] [<c01899e4>] (generic_handle_irq) from [<c018a088>] (__handle_domain_irq+0x5c/0xb4)
[  111.876798] [<c018a088>] (__handle_domain_irq) from [<c04feae8>] (gic_handle_irq+0x80/0x94)
[  111.885179] [<c04feae8>] (gic_handle_irq) from [<c0100b8c>] (__irq_svc+0x6c/0xa8)
[  111.892686] Exception stack(0xc1101ea0 to 0xc1101ee8)
[  111.897751] 1ea0: 00000000 c120e830 2e726000 ef793780 00000000 c120e830 00000000 00000000
[  111.905944] 1ec0: ef792938 c1254180 f4656888 00000019 fffffff6 c1101ef0 c0838164 c0838234
[  111.914130] 1ee0: 60000113 ffffffff
[  111.917624] [<c0100b8c>] (__irq_svc) from [<c0838234>] (cpuidle_enter_state+0x1c4/0x524)
[  111.925739] [<c0838234>] (cpuidle_enter_state) from [<c083a9cc>] (cpuidle_enter_state_coupled+0x144/0x3c8)
[  111.935419] [<c083a9cc>] (cpuidle_enter_state_coupled) from [<c08385f8>] (cpuidle_enter+0x50/0x54)
[  111.944402] [<c08385f8>] (cpuidle_enter) from [<c015aad0>] (do_idle+0x204/0x280)
[  111.951823] [<c015aad0>] (do_idle) from [<c015aea4>] (cpu_startup_entry+0x18/0x1c)
[  111.959414] [<c015aea4>] (cpu_startup_entry) from [<c1000e70>] (start_kernel+0x530/0x574)
[  111.966314] SMP: failed to stop secondary CPUs
[  111.972083] ---[ end Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. ]---

U-Boot SPL 2019.07-g2e30fa4be2 (Jan 12 2021 - 10:24:18 -0800) Trying to boot from RAM

U-Boot 2019.07-g2e30fa4be2 (Jan 12 2021 - 10:24:18 -0800)

TEGRA30

Model: NVIDIA Cardhu
Board: NVIDIA Cardhu
DRAM:  1 GiB
iBC:   sdhci@78000000: 1, sdhci@78000600: 0

Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  2  1  0 

switch to partitions #0, OK
mmc1 is current device
** No partition table - mmc 1 **
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
413 bytes read in 20 ms (19.5 KiB/s)
Cardhu NFS boot options
1: primary kernel
Enter choice: OIIIIIIIII)�k���՝M���!��ѥ���%��ѥ���镑�������)���������������������������������������������������5)9YI5�%��ѥ���镑�͡������х��͕)9YI5� 1= -M�A11a�遁�����������-��)9YI5� 1= -M�A115�遁�����������-��)9YI5� 1= -M�A11 �遁�����������-��)9YI5� 1= -M�A11A�遁�����������-��)9YI5� 1= -M�A11�遁���������-��)9YI5� 1= -M� AU遁�������������-��)9YI5� 1= -M�YA遁�������������-��)9YI5� 1= -M�M��ѕ�� ����������-��)9YI5� 1= -M�5����� ���ɽ�����������)9YI5� 1= -M��ѕɹ���5����� ���ɽ�����������)�The Bus seg 0:0x06 BoardInfo: 0x0c5b:0x0b11:0x04:0x43:0x03
The Bus seg 0:0x07 BoardInfo: 0x0c61:0x0a00:0x02:0x43:0x03 ADJUSTED CLOCKS:
MC clock is set to 400000 KHz
EMC clock is set to 800000 KHz (DDR clock is at 800000 KHz) PLLX0 clock is set to 700000 KHz
PLLC0 clock is set to 600000 KHz
CPU clock is set to 700000 KHz
System and AVP clock is set to 102000 KHz GraphicsHost clock is set to 163200 KHz
3D clock is set to 133333 KHz
2D clock is set to 133333 KHz
Epp clock is set to 133333 KHz
Mpe clock is set to 133333 KHz
Vde clock is set to 272000 KHz
Bootloader Start at:38875 ms

[bootloader] (built on Oct 21 2014, 10:28:12)
Initializing Display
Invalidate-only cache maint not supported in NvOs
Platform Pre Boot configuration...
Entering NvFlash recovery mode / Nv3p Server

Region=1 SD Erase start 512B-sector=0,512B-sector-num=2048 Region=2 SD Erase start 512B-sector=0,512B-sector-num=2048 Region=0 SD Erase start 512B-sector=0,512B-sector-num=31105024 LCM of 1024 and 4096 =4096
SD Alloc Partid=2, start sector=0,num=512
SD Alloc Partid=3, start sector=512,num=2048
SD Alloc Partid=4, start sector=2560,num=512
SD Alloc Partid=5, start sector=3072,num=1024
SD Alloc Partid=6, start sector=4096,num=4096
SD Alloc Partid=7, start sector=8192,num=1536
SD Alloc Partid=8, start sector=9728,num=512
SD Alloc Partid=9, start sector=10240,num=1536
SD Alloc Partid=10, start sector=11776,num=1536
SD Alloc Partid=11, start sector=13312,num=512
SD Alloc Partid=12, start sector=13824,num=3670016 SD Alloc Partid=13, start sector=3683840,num=1024
SD Alloc Partid=14, start sector=3684864,num=16384 SD Alloc Partid=15, start sector=3701248,num=1024
SD Alloc Partid=16, start sector=3702272,num=1024
SD Alloc Partid=17, start sector=3703296,num=1024
SD Alloc Partid=18, start sector=3704320,num=1024
SD Alloc Partid=19, start sector=3705344,num=512
SD Alloc Partid=20, start sector=3705856,num=182272 SD Alloc Partid=21, start sector=3888128,num=512 Region=0 SD Erase start 512B-sector=16384,512B-sector-num=4096 Start Downloading PPT

End Downloading PPT

Start Downloading EBT

End Downloading EBT

Start Downloading APP

End Downloading APP

Start Downloading DTB

End Downloading DTB

Start Downloading GPT

End Downloading GPT

U-Boot SPL 2019.07-g2e30fa4be2 (Jan 12 2021 - 10:29:58 -0800) Trying to boot from RAM

U-Boot 2019.07-g2e30fa4be2 (Jan 12 2021 - 10:29:58 -0800)

TEGRA30

Model: NVIDIA Cardhu
Board: NVIDIA Cardhu
DRAM:  1 GiB
iBC:   sdhci@78000000: 1, sdhci@78000600: 0

Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  2 

U-Boot SPL 2019.07-g2e30fa4be2 (Jan 12 2021 - 10:29:58 -0800) Trying to boot from RAM

U-Boot 2019.07-g2e30fa4be2 (Jan 12 2021 - 10:29:58 -0800)

TEGRA30

Model: NVIDIA Cardhu
Board: NVIDIA Cardhu
DRAM:  1 GiB
iBC:   sdhci@78000000: 1, sdhci@78000600: 0

Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  2  1  0 

switch to partitions #0, OK
mmc1 is current device
** No partition table - mmc 1 **
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
413 bytes read in 20 ms (19.5 KiB/s)
Cardhu NFS boot options
1: primary kernel
Enter choice: 1: primary kernel
Retrieving file: /boot/zImage
9703936 bytes read in 351 ms (26.4 MiB/s)
append: console=ttyS0,115200n8 console=tty1 earlycon earlyprintk ignore_loglevel root=/dev/nfs rw netdevwait ip=192.168.99.2:192.168.99.1:192.168.99.1:255.255.255.0::eth0:off nfsroot=192.168.99.1:/home/ausvrl2777/nfsroot,tcp rootwait Retrieving file: /boot/tegra30-cardhu-a04.dtb
44584 bytes read in 20 ms (2.1 MiB/s)
## Flattened Device Tree blob at 83000000
Booting using the fdt blob at 0x83000000
Using Device Tree in place at 83000000, end 8300de27

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.11.0-rc3-next-20210112-gdf869cab4b35 (buildbrain@mobile-u64-5252-d8000) (arm-linux-gnueabihf-gcc (Linaro GCC 7.3-2018.05) 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701], GNU ld (Linaro_Binutils-2018.05) 2.28.2.20170706) #1 SMP Tue Jan 12 10:30:14 PST 2021
[    0.000000] CPU: ARMv7 Processor [412fc099] revision 9 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: NVIDIA Tegra30 Cardhu A04 (A05, A06, A07) evaluation board
[    0.000000] earlycon: uart0 at MMIO 0x70006000 (options '115200n8')
[    0.000000] printk: bootconsole [uart0] enabled
[    0.000000] printk: debug: ignoring loglevel setting.
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 64 MiB at 0xbc000000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x00000000b0000000-0x00000000bfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] On node 0 totalpages: 262144
[    0.000000]   DMA zone: 1536 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 196608 pages, LIFO batch:63
[    0.000000]   HighMem zone: 65536 pages, LIFO batch:15
[    0.000000] percpu: Embedded 20 pages/cpu s49996 r8192 d23732 u81920
[    0.000000] pcpu-alloc: s49996 r8192 d23732 u81920 alloc=20*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 260608
[    0.000000] Kernel command line: console=ttyS0,115200n8 console=tty1 earlycon earlyprintk ignore_loglevel root=/dev/nfs rw netdevwait ip=192.168.99.2:192.168.99.1:192.168.99.1:255.255.255.0::eth0:off nfsroot=192.168.99.1:/home/ausvrl2777/nfsroot,tcp rootwait
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 949452K/1048576K available (13312K kernel code, 2183K rwdata, 5360K rodata, 2048K init, 401K bss, 33588K reserved, 65536K cma-reserved, 196608K highmem)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=4.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] /interrupt-controller@60004000: 160 interrupts forwarded to /interrupt-controller@50041000
[    0.000000] L2C: platform modifies aux control register: 0x02080000 -> 0x3e480000
[    0.000000] L2C: DT/platform modifies aux control register: 0x02080000 -> 0x3e480000
[    0.000000] L2C-310 erratum 769419 enabled
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[    0.000000] L2C-310 cache controller enabled, 8 ways, 1024 kB
[    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e480001
[    0.000000] random: get_random_bytes called from start_kernel+0x3b4/0x558 with crng_init=0
[    0.000002] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[    0.017221] clocksource: timer_us: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[    0.027138] Switching to timer-based delay loop, resolution 1000ns
[    0.034446] clocksource: tegra_suspend_timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.049813] Console: colour dummy device 80x30
[    0.054437] printk: console [tty1] enabled
[    0.058688] printk: bootconsole [uart0] disabled
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.11.0-rc3-next-20210112-gdf869cab4b35 (buildbrain@mobile-u64-5252-d8000) (arm-linux-gnueabihf-gcc (Linaro GCC 7.3-2018.05) 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701], GNU ld (Linaro_Binutils-2018.05) 2.28.2.20170706) #1 SMP Tue Jan 12 10:30:14 PST 2021
[    0.000000] CPU: ARMv7 Processor [412fc099] revision 9 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: NVIDIA Tegra30 Cardhu A04 (A05, A06, A07) evaluation board
[    0.000000] earlycon: uart0 at MMIO 0x70006000 (options '115200n8')
[    0.000000] printk: bootconsole [uart0] enabled
[    0.000000] printk: debug: ignoring loglevel setting.
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 64 MiB at 0xbc000000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x00000000b0000000-0x00000000bfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] On node 0 totalpages: 262144
[    0.000000]   DMA zone: 1536 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 196608 pages, LIFO batch:63
[    0.000000]   HighMem zone: 65536 pages, LIFO batch:15
[    0.000000] percpu: Embedded 20 pages/cpu s49996 r8192 d23732 u81920
[    0.000000] pcpu-alloc: s49996 r8192 d23732 u81920 alloc=20*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 260608
[    0.000000] Kernel command line: console=ttyS0,115200n8 console=tty1 earlycon earlyprintk ignore_loglevel root=/dev/nfs rw netdevwait ip=192.168.99.2:192.168.99.1:192.168.99.1:255.255.255.0::eth0:off nfsroot=192.168.99.1:/home/ausvrl2777/nfsroot,tcp rootwait
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 949452K/1048576K available (13312K kernel code, 2183K rwdata, 5360K rodata, 2048K init, 401K bss, 33588K reserved, 65536K cma-reserved, 196608K highmem)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=4.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] /interrupt-controller@60004000: 160 interrupts forwarded to /interrupt-controller@50041000
[    0.000000] L2C: platform modifies aux control register: 0x02080000 -> 0x3e480000
[    0.000000] L2C: DT/platform modifies aux control register: 0x02080000 -> 0x3e480000
[    0.000000] L2C-310 erratum 769419 enabled
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[    0.000000] L2C-310 cache controller enabled, 8 ways, 1024 kB
[    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e480001
[    0.000000] random: get_random_bytes called from start_kernel+0x3b4/0x558 with crng_init=0
[    0.000002] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[    0.017221] clocksource: timer_us: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[    0.027138] Switching to timer-based delay loop, resolution 1000ns
[    0.034446] clocksource: tegra_suspend_timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.049813] Console: colour dummy device 80x30
[    0.054437] printk: console [tty1] enabled
[    0.058688] printk: bootconsole [uart0] disabled
[    0.063522] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=10000)
[    0.063575] pid_max: default: 32768 minimum: 301
[    0.063785] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[    0.063826] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[    0.064631] CPU: Testing write buffer coherency: ok
[    0.064695] CPU0: Spectre v2: using BPIALL workaround
[    0.065026] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.065663] Setting up static identity map for 0x80300000 - 0x803000ac
[    0.067782] rcu: Hierarchical SRCU implementation.
[    0.071523] Tegra Revision: A02 SKU: 129 CPU Process: 2 SoC Process: 0
[    0.072446] EFI services will not be available.
[    0.072705] smp: Bringing up secondary CPUs ...
[    0.073930] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.073948] CPU1: Spectre v2: using BPIALL workaround
[    0.083941] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[    0.083958] CPU2: Spectre v2: using BPIALL workaround
[    0.093914] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[    0.093935] CPU3: Spectre v2: using BPIALL workaround
[    0.094090] smp: Brought up 1 node, 4 CPUs
[    0.094120] SMP: Total of 4 processors activated (8.00 BogoMIPS).
[    0.094145] CPU: All CPU(s) started in SVC mode.
[    0.094804] devtmpfs: initialized
[    0.105821] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[    0.106115] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.106176] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.110613] pinctrl core: initialized pinctrl subsystem
[    0.112990] DMI not present or invalid.
[    0.113440] NET: Registered protocol family 16
[    0.117296] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.120375] thermal_sys: Registered thermal governor 'step_wise'
[    0.120621] cpuidle: using governor menu
[    0.150402] No ATAGs?
[    0.150600] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[    0.150642] hw-breakpoint: maximum watchpoint size is 4 bytes.
[    0.161254] Serial: AMBA PL011 UART driver
[    0.162352] tegra-mc 7000f000.memory-controller: no memory timings for RAM code 0 registered
[    0.189679] iommu: Default domain type: Translated 
[    0.189986] vgaarb: loaded
[    0.190926] SCSI subsystem initialized
[    0.191155] libata version 3.00 loaded.
[    0.191471] usbcore: registered new interface driver usbfs
[    0.191561] usbcore: registered new interface driver hub
[    0.191633] usbcore: registered new device driver usb
[    0.193291] pps_core: LinuxPPS API ver. 1 registered
[    0.193319] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.193370] PTP clock support registered
[    0.196789] clocksource: Switched to clocksource timer_us
[    0.250777] NET: Registered protocol family 2
[    1.346772] random: fast init done
[    1.781698] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[    1.781773] TCP established hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    1.781927] TCP bind hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    1.782117] TCP: Hash tables configured (established 8192 bind 8192)
[    1.782283] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[    1.782376] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[    1.782638] NET: Registered protocol family 1
[    1.783271] RPC: Registered named UNIX socket transport module.
[    1.783308] RPC: Registered udp transport module.
[    1.783331] RPC: Registered tcp transport module.
[    1.783351] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.783379] PCI: CLS 0 bytes, default 64
[    1.785049] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[    1.786481] Initialise system trusted keyrings
[    1.786730] workingset: timestamp_bits=30 max_order=18 bucket_order=0
[    1.794324] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.795179] NFS: Registering the id_resolver key type
[    1.795229] Key type id_resolver registered
[    1.795254] Key type id_legacy registered
[    1.795394] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.795457] ntfs: driver 2.1.32 [Flags: R/O].
[    1.795967] Key type asymmetric registered
[    1.795998] Asymmetric key parser 'x509' registered
[    1.796085] bounce: pool size: 64 pages
[    1.796141] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    1.796174] io scheduler mq-deadline registered
[    1.796198] io scheduler kyber registered
[    1.832154] tegra-apbdma 6000a000.dma: Tegra20 APB DMA driver registered 32 channels
[    1.838450] tegra-pmc 7000e400.pmc: i2c-thermtrip node not found, emergency thermal reset disabled.
[    1.906724] Serial: 8250/16550 driver, 5 ports, IRQ sharing enabled
[    1.911417] printk: console [ttyS0] disabled
[    1.911536] 70006000.serial: ttyS0 at MMIO 0x70006000 (irq = 79, base_baud = 25500000) is a Tegra
[    2.751343] printk: console [ttyS0] enabled
[    2.757083] SuperH (H)SCI(F) driver initialized
[    2.762923] msm_serial: driver initialized
[    2.767293] STMicroelectronics ASC driver initialized
[    2.773524] 70006200.serial: ttyTHS1 at MMIO 0x70006200 (irq = 80, base_baud = 0) is a TEGRA_UART
[    2.783538] STM32 USART driver initialized
[    2.790792] tegra-host1x 50000000.host1x: Adding to iommu group 0
[    2.813029] tegra-gr2d 54140000.gr2d: Adding to iommu group 1
[    2.819402] tegra-gr3d 54180000.gr3d: Adding to iommu group 1
[    2.825188] Failed to attached device 54180000.gr3d to IOMMU_mapping
[    2.846185] brd: module loaded
[    2.859126] loop: module loaded
[    2.876094] spi-nor spi0.1: w25q32 (4096 Kbytes)
[    2.884964] libphy: Fixed MDIO Bus: probed
[    2.891354] CAN device driver interface
[    2.896550] bgmac_bcma: Broadcom 47xx GBit MAC driver loaded
[    2.903828] e1000e: Intel(R) PRO/1000 Network Driver
[    2.908827] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    2.914837] igb: Intel(R) Gigabit Ethernet Network Driver
[    2.920266] igb: Copyright (c) 2007-2014 Intel Corporation.
[    2.932097] pegasus: v0.9.3 (2013/04/25), Pegasus/Pegasus II USB Ethernet driver
[    2.939588] usbcore: registered new interface driver pegasus
[    2.945321] usbcore: registered new interface driver asix
[    2.950796] usbcore: registered new interface driver ax88179_178a
[    2.956969] usbcore: registered new interface driver cdc_ether
[    2.962864] usbcore: registered new interface driver smsc75xx
[    2.968697] usbcore: registered new interface driver smsc95xx
[    2.974490] usbcore: registered new interface driver net1080
[    2.980225] usbcore: registered new interface driver cdc_subset
[    2.986190] usbcore: registered new interface driver zaurus
[    2.991862] usbcore: registered new interface driver cdc_ncm
[    3.001154] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    3.007720] ehci-pci: EHCI PCI platform driver
[    3.012220] ehci-platform: EHCI generic platform driver
[    3.017771] ehci-orion: EHCI orion driver
[    3.022009] SPEAr-ehci: EHCI SPEAr driver
[    3.026225] ehci-st: EHCI STMicroelectronics driver
[    3.031334] ehci-atmel: EHCI Atmel driver
[    3.035537] tegra-ehci: Tegra EHCI driver
[    3.039993] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    3.046198] ohci-pci: OHCI PCI platform driver
[    3.050731] ohci-platform: OHCI generic platform driver
[    3.056224] SPEAr-ohci: OHCI SPEAr driver
[    3.060464] ohci-st: OHCI STMicroelectronics driver
[    3.065540] ohci-atmel: OHCI Atmel driver
[    3.070450] usbcore: registered new interface driver usb-storage
[    3.078711] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.090219] tegra_rtc 7000e000.rtc: registered as rtc1
[    3.095378] tegra_rtc 7000e000.rtc: Tegra internal Real Time Clock
[    3.102449] i2c /dev entries driver
[    3.120241] i2c i2c-2: Added multiplexed i2c bus 5
[    3.125369] i2c i2c-2: Added multiplexed i2c bus 6
[    3.130530] i2c i2c-2: Added multiplexed i2c bus 7
[    3.135634] i2c i2c-2: Added multiplexed i2c bus 8
[    3.140483] pca954x 2-0070: registered 4 multiplexed busses for I2C switch pca9546
[    3.162699] sdhci: Secure Digital Host Controller Interface driver
[    3.168927] sdhci: Copyright(c) Pierre Ossman
[    3.176005] Synopsys Designware Multimedia Card Interface Driver
[    3.184123] sdhci-pltfm: SDHCI platform and OF driver helper
[    3.191362] sdhci-tegra 78000000.mmc: Got CD GPIO
[    3.196131] sdhci-tegra 78000000.mmc: Got WP GPIO
[    3.201111] ledtrig-cpu: registered to indicate activity on CPUs
[    3.205537] mmc1: Invalid maximum block size, assuming 512 bytes
[    3.208271] usbcore: registered new interface driver usbhid
[    3.209173] mmc0: Invalid maximum block size, assuming 512 bytes
[    3.213346] mmc2: Invalid maximum block size, assuming 512 bytes
[    3.218786] usbhid: USB HID core driver
[    3.238591] tegra30-emc 7000f400.memory-controller: device-tree doesn't have memory timings
[    3.243914] mmc0: SDHCI controller on 78000000.mmc [78000000.mmc] using ADMA
[    3.249174] tegra30-emc 7000f400.memory-controller: OPP HW ver. 0x4, current clock rate 800 MHz
[    3.256835] mmc1: SDHCI controller on 78000400.mmc [78000400.mmc] using ADMA
[    3.264297] isl29028 2-0044: No cache defaults, reading back from HW
[    3.269992] mmc2: SDHCI controller on 78000600.mmc [78000600.mmc] using ADMA
[    3.286929] NET: Registered protocol family 10
[    3.292593] Segment Routing with IPv6
[    3.296354] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    3.303000] NET: Registered protocol family 17
[    3.307503] can: controller area network core
[    3.311942] NET: Registered protocol family 29
[    3.316399] can: raw protocol
[    3.319417] can: broadcast manager protocol
[    3.323614] can: netlink gateway - max_hops=1
[    3.328482] Key type dns_resolver registered
[    3.335211] tegra20-cpufreq tegra20-cpufreq: hardware version 0x4 0x4
[    3.342527] ThumbEE CPU extension supported.
[    3.346852] Registering SWP/SWPB emulation handler
[    3.352144] Loading compiled-in X.509 certificates
[    3.398765] tegra-dc 54200000.dc: Adding to iommu group 1
[    3.404239] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.416892] tegra-dc 54200000.dc: failed to probe RGB output: -517
[    3.425249] tegra-dc 54240000.dc: Adding to iommu group 1
[    3.430772] Failed to attached device 54240000.dc to IOMMU_mapping
[    3.440576] mmc0: new high speed SDHC card at address e624
[    3.446932] mmcblk0: mmc0:e624 SD08G 7.40 GiB 
[    3.449196] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.472368] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.486876] tegra-dc 54200000.dc: failed to probe RGB output: -517
[    3.497746] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.517835] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.525467] mmc2: new high speed MMC card at address 0001
[    3.530900] tegra-dc 54200000.dc: failed to probe RGB output: -517
[    3.535026] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.538221] mmcblk2: mmc2:0001 SEM16G 14.8 GiB 
[    3.547674] mmcblk2boot0: mmc2:0001 SEM16G partition 1 1.00 MiB
[    3.553984] mmcblk2boot1: mmc2:0001 SEM16G partition 2 1.00 MiB
[    3.558191] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.560285] mmcblk2rpmb: mmc2:0001 SEM16G partition 3 128 KiB, chardev (235:0)
[    3.574346] tegra-dc 54200000.dc: failed to probe RGB output: -517
[    3.582607] irq: no irq domain found for tps65911@2d !
[    3.587915] gpio-keys gpio-keys: Found button without gpio or irq
[    3.594040] gpio-keys: probe of gpio-keys failed with error -22
[    3.605331] Alternate GPT is invalid, using primary GPT.
[    3.610739]  mmcblk2: p1 p2 p3 p4 p5 p6 p7 p8 p9
[    3.624942] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.645352] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.659778] tegra-dc 54200000.dc: failed to probe RGB output: -517
[   46.599366] modem_3v3: disabling
[  111.879731] VFS: Unable to mount root fs via NFS.
[  111.884501] devtmpfs: mounted
[  111.890451] Freeing unused kernel memory: 2048K
[  111.908272] Run /sbin/init as init process
[  111.912388]   with arguments:
[  111.915358]     /sbin/init
[  111.918098]     earlyprintk
[  111.920902]     netdevwait
[  111.923612]   with environment:
[  111.926754]     HOME=/
[  111.929139]     TERM=linux
[  111.932096] Run /etc/init as init process
[  111.936120]   with arguments:
[  111.939113]     /etc/init
[  111.941741]     earlyprintk
[  111.944538]     netdevwait
[  111.947278]   with environment:
[  111.950426]     HOME=/
[  111.952791]     TERM=linux
[  111.955617] Run /bin/init as init process
[  111.959667]   with arguments:
[  111.962644]     /bin/init
[  111.965268]     earlyprintk
[  111.968097]     netdevwait
[  111.970812]   with environment:
[  111.973956]     HOME=/
[  111.976317]     TERM=linux
[  111.979162] Run /bin/sh as init process
[  111.983009]   with arguments:
[  111.985979]     /bin/sh
[  111.988456]     earlyprintk
[  111.991261]     netdevwait
[  111.993971]   with environment:
[  111.997149]     HOME=/
[  111.999521]     TERM=linux
[  112.002347] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[  112.016531] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  112.025248] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  112.031535] [<c03114f8>] (unwind_backtrace) from [<c030bcec>] (show_stack+0x10/0x14)
[  112.039330] [<c030bcec>] (show_stack) from [<c0fc129c>] (dump_stack+0xc8/0xdc)
[  112.046589] [<c0fc129c>] (dump_stack) from [<c0fbfb60>] (panic+0x108/0x320)
[  112.053572] [<c0fbfb60>] (panic) from [<c0fc7540>] (kernel_init+0x108/0x118)
[  112.060652] [<c0fc7540>] (kernel_init) from [<c03001b0>] (ret_from_fork+0x14/0x24)
[  112.068250] Exception stack(0xc20edfb0 to 0xc20edff8)
[  112.073316] dfa0:                                     00000000 00000000 00000000 00000000
[  112.081503] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  112.089688] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  112.096318] CPU2: stopping
[  112.099040] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  112.107751] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  112.114022] [<c03114f8>] (unwind_backtrace) from [<c030bcec>] (show_stack+0x10/0x14)
[  112.121792] [<c030bcec>] (show_stack) from [<c0fc129c>] (dump_stack+0xc8/0xdc)
[  112.129035] [<c0fc129c>] (dump_stack) from [<c030ef48>] (do_handle_IPI+0x320/0x33c)
[  112.136712] [<c030ef48>] (do_handle_IPI) from [<c030ef7c>] (ipi_handler+0x18/0x20)
[  112.144300] [<c030ef7c>] (ipi_handler) from [<c03a7b4c>] (handle_percpu_devid_irq+0x8c/0x21c)
[  112.152850] [<c03a7b4c>] (handle_percpu_devid_irq) from [<c03a16a8>] (generic_handle_irq+0x34/0x44)
[  112.161924] [<c03a16a8>] (generic_handle_irq) from [<c03a1d64>] (__handle_domain_irq+0x5c/0xb4)
[  112.170644] [<c03a1d64>] (__handle_domain_irq) from [<c0768968>] (gic_handle_irq+0x94/0xb4)
[  112.179019] [<c0768968>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
[  112.186518] Exception stack(0xc2113ee0 to 0xc2113f28)
[  112.191580] 3ee0: 00000000 c19e8850 2e065000 ef7ba7c0 00000000 c19e8850 00000000 00000000
[  112.199767] 3f00: ef7b9830 c1a13560 1975fea8 0000001a fffffff6 c2113f30 c0d4b35c c0d4b42c
[  112.207949] 3f20: 60000113 ffffffff
[  112.211441] [<c0300b8c>] (__irq_svc) from [<c0d4b42c>] (cpuidle_enter_state+0x1c4/0x4ac)
[  112.219569] [<c0d4b42c>] (cpuidle_enter_state) from [<c0d4dd58>] (cpuidle_enter_state_coupled+0x144/0x3c8)
[  112.229247] [<c0d4dd58>] (cpuidle_enter_state_coupled) from [<c0d4b778>] (cpuidle_enter+0x50/0x54)
[  112.238221] [<c0d4b778>] (cpuidle_enter) from [<c03793c0>] (do_idle+0x204/0x280)
[  112.245640] [<c03793c0>] (do_idle) from [<c0379748>] (cpu_startup_entry+0x18/0x1c)
[  112.253229] [<c0379748>] (cpu_startup_entry) from [<80301750>] (0x80301750)
[  112.260204] CPU3: stopping
[  112.262927] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  112.271638] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  112.277908] [<c03114f8>] (unwind_backtrace) from [<c030bcec>] (show_stack+0x10/0x14)
[  112.285680] [<c030bcec>] (show_stack) from [<c0fc129c>] (dump_stack+0xc8/0xdc)
[  112.292923] [<c0fc129c>] (dump_stack) from [<c030ef48>] (do_handle_IPI+0x320/0x33c)
[  112.300599] [<c030ef48>] (do_handle_IPI) from [<c030ef7c>] (ipi_handler+0x18/0x20)
[  112.308187] [<c030ef7c>] (ipi_handler) from [<c03a7b4c>] (handle_percpu_devid_irq+0x8c/0x21c)
[  112.316732] [<c03a7b4c>] (handle_percpu_devid_irq) from [<c03a16a8>] (generic_handle_irq+0x34/0x44)
[  112.325799] [<c03a16a8>] (generic_handle_irq) from [<c03a1d64>] (__handle_domain_irq+0x5c/0xb4)
[  112.334520] [<c03a1d64>] (__handle_domain_irq) from [<c0768968>] (gic_handle_irq+0x94/0xb4)
[  112.342890] [<c0768968>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
[  112.350389] Exception stack(0xc211bee0 to 0xc211bf28)
[  112.355453] bee0: 00000000 c19e8850 2e079000 ef7ce7c0 00000000 c19e8850 00000000 00000000
[  112.363640] bf00: ef7cd830 c1a13560 1975fea8 0000001a fffffff6 c211bf30 c0d4b35c c0d4b42c
[  112.371823] bf20: 60000113 ffffffff
[  112.375315] [<c0300b8c>] (__irq_svc) from [<c0d4b42c>] (cpuidle_enter_state+0x1c4/0x4ac)
[  112.383431] [<c0d4b42c>] (cpuidle_enter_state) from [<c0d4dd58>] (cpuidle_enter_state_coupled+0x144/0x3c8)
[  112.393107] [<c0d4dd58>] (cpuidle_enter_state_coupled) from [<c0d4b778>] (cpuidle_enter+0x50/0x54)
[  112.402081] [<c0d4b778>] (cpuidle_enter) from [<c03793c0>] (do_idle+0x204/0x280)
[  112.409495] [<c03793c0>] (do_idle) from [<c0379748>] (cpu_startup_entry+0x18/0x1c)
[  112.417083] [<c0379748>] (cpu_startup_entry) from [<80301750>] (0x80301750)
[  112.424058] CPU1: stopping
[  112.426781] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  112.435491] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  112.441761] [<c03114f8>] (unwind_backtrace) from [<c030bcec>] (show_stack+0x10/0x14)
[  112.449534] [<c030bcec>] (show_stack) from [<c0fc129c>] (dump_stack+0xc8/0xdc)
[  112.456777] [<c0fc129c>] (dump_stack) from [<c030ef48>] (do_handle_IPI+0x320/0x33c)
[  112.464452] [<c030ef48>] (do_handle_IPI) from [<c030ef7c>] (ipi_handler+0x18/0x20)
[  112.472041] [<c030ef7c>] (ipi_handler) from [<c03a7b4c>] (handle_percpu_devid_irq+0x8c/0x21c)
[  112.480586] [<c03a7b4c>] (handle_percpu_devid_irq) from [<c03a16a8>] (generic_handle_irq+0x34/0x44)
[  112.489651] [<c03a16a8>] (generic_handle_irq) from [<c03a1d64>] (__handle_domain_irq+0x5c/0xb4)
[  112.498371] [<c03a1d64>] (__handle_domain_irq) from [<c0768968>] (gic_handle_irq+0x94/0xb4)
[  112.506741] [<c0768968>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
[  112.514241] Exception stack(0xc2111ee0 to 0xc2111f28)
[  112.519304] 1ee0: 00000000 c19e8850 2e051000 ef7a67c0 00000000 c19e8850 00000000 00000000
[  112.527491] 1f00: ef7a5830 c1a13560 1975fea8 0000001a fffffff6 c2111f30 c0d4b35c c0d4b42c
[  112.535672] 1f20: 60000113 ffffffff
[  112.539164] [<c0300b8c>] (__irq_svc) from [<c0d4b42c>] (cpuidle_enter_state+0x1c4/0x4ac)
[  112.547281] [<c0d4b42c>] (cpuidle_enter_state) from [<c0d4dd58>] (cpuidle_enter_state_coupled+0x144/0x3c8)
[  112.556957] [<c0d4dd58>] (cpuidle_enter_state_coupled) from [<c0d4b778>] (cpuidle_enter+0x50/0x54)
[  112.565932] [<c0d4b778>] (cpuidle_enter) from [<c03793c0>] (do_idle+0x204/0x280)
[  112.573346] [<c03793c0>] (do_idle) from [<c0379748>] (cpu_startup_entry+0x18/0x1c)
[  112.580933] [<c0379748>] (cpu_startup_entry) from [<80301750>] (0x80301750)
[  112.587922] ---[ end Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. ]---

U-Boot SPL 2019.07-g2e30fa4be2 (Jan 12 2021 - 10:29:58 -0800) Trying to boot from RAM

U-Boot 2019.07-g2e30fa4be2 (Jan 12 2021 - 10:29:58 -0800)

TEGRA30

Model: NVIDIA Cardhu
Board: NVIDIA Cardhu
DRAM:  1 GiB
iBC:   sdhci@78000000: 1, sdhci@78000600: 0

Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  2  1  0 

switch to partitions #0, OK
mmc1 is current device
** No partition table - mmc 1 **
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
413 bytes read in 20 ms (19.5 KiB/s)
Cardhu NFS boot options
1: primary kernel
Enter choice: 1: primary kernel
Retrieving file: /boot/zImage
9703936 bytes read in 350 ms (26.4 MiB/s)
append: console=ttyS0,115200n8 console=tty1 earlycon earlyprintk ignore_loglevel root=/dev/nfs rw netdevwait ip=192.168.99.2:192.168.99.1:192.168.99.1:255.255.255.0::eth0:off nfsroot=192.168.99.1:/home/ausvrl2777/nfsroot,tcp rootwait Retrieving file: /boot/tegra30-cardhu-a04.dtb
44584 bytes read in 21 ms (2 MiB/s)
## Flattened Device Tree blob at 83000000
Booting using the fdt blob at 0x83000000
Using Device Tree in place at 83000000, end 8300de27

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.11.0-rc3-next-20210112-gdf869cab4b35 (buildbrain@mobile-u64-5252-d8000) (arm-linux-gnueabihf-gcc (Linaro GCC 7.3-2018.05) 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701], GNU ld (Linaro_Binutils-2018.05) 2.28.2.20170706) #1 SMP Tue Jan 12 10:30:14 PST 2021
[    0.000000] CPU: ARMv7 Processor [412fc099] revision 9 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: NVIDIA Tegra30 Cardhu A04 (A05, A06, A07) evaluation board
[    0.000000] earlycon: uart0 at MMIO 0x70006000 (options '115200n8')
[    0.000000] printk: bootconsole [uart0] enabled
[    0.000000] printk: debug: ignoring loglevel setting.
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 64 MiB at 0xbc000000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x00000000b0000000-0x00000000bfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] On node 0 totalpages: 262144
[    0.000000]   DMA zone: 1536 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 196608 pages, LIFO batch:63
[    0.000000]   HighMem zone: 65536 pages, LIFO batch:15
[    0.000000] percpu: Embedded 20 pages/cpu s49996 r8192 d23732 u81920
[    0.000000] pcpu-alloc: s49996 r8192 d23732 u81920 alloc=20*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 260608
[    0.000000] Kernel command line: console=ttyS0,115200n8 console=tty1 earlycon earlyprintk ignore_loglevel root=/dev/nfs rw netdevwait ip=192.168.99.2:192.168.99.1:192.168.99.1:255.255.255.0::eth0:off nfsroot=192.168.99.1:/home/ausvrl2777/nfsroot,tcp rootwait
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 949452K/1048576K available (13312K kernel code, 2183K rwdata, 5360K rodata, 2048K init, 401K bss, 33588K reserved, 65536K cma-reserved, 196608K highmem)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=4.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] /interrupt-controller@60004000: 160 interrupts forwarded to /interrupt-controller@50041000
[    0.000000] L2C: platform modifies aux control register: 0x02080000 -> 0x3e480000
[    0.000000] L2C: DT/platform modifies aux control register: 0x02080000 -> 0x3e480000
[    0.000000] L2C-310 erratum 769419 enabled
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[    0.000000] L2C-310 cache controller enabled, 8 ways, 1024 kB
[    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e480001
[    0.000000] random: get_random_bytes called from start_kernel+0x3b4/0x558 with crng_init=0
[    0.000002] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[    0.017217] clocksource: timer_us: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[    0.027133] Switching to timer-based delay loop, resolution 1000ns
[    0.034398] clocksource: tegra_suspend_timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.049739] Console: colour dummy device 80x30
[    0.054364] printk: console [tty1] enabled
[    0.058615] printk: bootconsole [uart0] disabled
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.11.0-rc3-next-20210112-gdf869cab4b35 (buildbrain@mobile-u64-5252-d8000) (arm-linux-gnueabihf-gcc (Linaro GCC 7.3-2018.05) 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701], GNU ld (Linaro_Binutils-2018.05) 2.28.2.20170706) #1 SMP Tue Jan 12 10:30:14 PST 2021
[    0.000000] CPU: ARMv7 Processor [412fc099] revision 9 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: NVIDIA Tegra30 Cardhu A04 (A05, A06, A07) evaluation board
[    0.000000] earlycon: uart0 at MMIO 0x70006000 (options '115200n8')
[    0.000000] printk: bootconsole [uart0] enabled
[    0.000000] printk: debug: ignoring loglevel setting.
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 64 MiB at 0xbc000000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x00000000b0000000-0x00000000bfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] On node 0 totalpages: 262144
[    0.000000]   DMA zone: 1536 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 196608 pages, LIFO batch:63
[    0.000000]   HighMem zone: 65536 pages, LIFO batch:15
[    0.000000] percpu: Embedded 20 pages/cpu s49996 r8192 d23732 u81920
[    0.000000] pcpu-alloc: s49996 r8192 d23732 u81920 alloc=20*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 260608
[    0.000000] Kernel command line: console=ttyS0,115200n8 console=tty1 earlycon earlyprintk ignore_loglevel root=/dev/nfs rw netdevwait ip=192.168.99.2:192.168.99.1:192.168.99.1:255.255.255.0::eth0:off nfsroot=192.168.99.1:/home/ausvrl2777/nfsroot,tcp rootwait
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 949452K/1048576K available (13312K kernel code, 2183K rwdata, 5360K rodata, 2048K init, 401K bss, 33588K reserved, 65536K cma-reserved, 196608K highmem)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=4.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] /interrupt-controller@60004000: 160 interrupts forwarded to /interrupt-controller@50041000
[    0.000000] L2C: platform modifies aux control register: 0x02080000 -> 0x3e480000
[    0.000000] L2C: DT/platform modifies aux control register: 0x02080000 -> 0x3e480000
[    0.000000] L2C-310 erratum 769419 enabled
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[    0.000000] L2C-310 cache controller enabled, 8 ways, 1024 kB
[    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e480001
[    0.000000] random: get_random_bytes called from start_kernel+0x3b4/0x558 with crng_init=0
[    0.000002] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[    0.017217] clocksource: timer_us: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[    0.027133] Switching to timer-based delay loop, resolution 1000ns
[    0.034398] clocksource: tegra_suspend_timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.049739] Console: colour dummy device 80x30
[    0.054364] printk: console [tty1] enabled
[    0.058615] printk: bootconsole [uart0] disabled
[    0.063448] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=10000)
[    0.063488] pid_max: default: 32768 minimum: 301
[    0.063717] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[    0.063758] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[    0.064563] CPU: Testing write buffer coherency: ok
[    0.064627] CPU0: Spectre v2: using BPIALL workaround
[    0.064956] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.065594] Setting up static identity map for 0x80300000 - 0x803000ac
[    0.067723] rcu: Hierarchical SRCU implementation.
[    0.071490] Tegra Revision: A02 SKU: 129 CPU Process: 2 SoC Process: 0
[    0.072399] EFI services will not be available.
[    0.072659] smp: Bringing up secondary CPUs ...
[    0.073925] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.073944] CPU1: Spectre v2: using BPIALL workaround
[    0.083930] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[    0.083949] CPU2: Spectre v2: using BPIALL workaround
[    0.093907] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[    0.093926] CPU3: Spectre v2: using BPIALL workaround
[    0.094088] smp: Brought up 1 node, 4 CPUs
[    0.094118] SMP: Total of 4 processors activated (8.00 BogoMIPS).
[    0.094143] CPU: All CPU(s) started in SVC mode.
[    0.094802] devtmpfs: initialized
[    0.105825] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[    0.106117] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.106177] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.110609] pinctrl core: initialized pinctrl subsystem
[    0.112998] DMI not present or invalid.
[    0.113450] NET: Registered protocol family 16
[    0.117471] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.120549] thermal_sys: Registered thermal governor 'step_wise'
[    0.120791] cpuidle: using governor menu
[    0.150564] No ATAGs?
[    0.150762] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[    0.150803] hw-breakpoint: maximum watchpoint size is 4 bytes.
[    0.161389] Serial: AMBA PL011 UART driver
[    0.162484] tegra-mc 7000f000.memory-controller: no memory timings for RAM code 0 registered
[    0.189755] iommu: Default domain type: Translated 
[    0.190061] vgaarb: loaded
[    0.191021] SCSI subsystem initialized
[    0.191249] libata version 3.00 loaded.
[    0.191565] usbcore: registered new interface driver usbfs
[    0.191656] usbcore: registered new interface driver hub
[    0.191730] usbcore: registered new device driver usb
[    0.193398] pps_core: LinuxPPS API ver. 1 registered
[    0.193427] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.193478] PTP clock support registered
[    0.196893] clocksource: Switched to clocksource timer_us
[    0.250890] NET: Registered protocol family 2
[    1.346877] random: fast init done
[    1.782408] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[    1.782485] TCP established hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    1.782616] TCP bind hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    1.782806] TCP: Hash tables configured (established 8192 bind 8192)
[    1.782974] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[    1.783064] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[    1.783323] NET: Registered protocol family 1
[    1.784064] RPC: Registered named UNIX socket transport module.
[    1.784103] RPC: Registered udp transport module.
[    1.784126] RPC: Registered tcp transport module.
[    1.784147] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.784176] PCI: CLS 0 bytes, default 64
[    1.786011] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[    1.787516] Initialise system trusted keyrings
[    1.787726] workingset: timestamp_bits=30 max_order=18 bucket_order=0
[    1.795228] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.796083] NFS: Registering the id_resolver key type
[    1.796132] Key type id_resolver registered
[    1.796157] Key type id_legacy registered
[    1.796294] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.796358] ntfs: driver 2.1.32 [Flags: R/O].
[    1.796930] Key type asymmetric registered
[    1.796962] Asymmetric key parser 'x509' registered
[    1.797048] bounce: pool size: 64 pages
[    1.797108] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    1.797142] io scheduler mq-deadline registered
[    1.797164] io scheduler kyber registered
[    1.832843] tegra-apbdma 6000a000.dma: Tegra20 APB DMA driver registered 32 channels
[    1.839130] tegra-pmc 7000e400.pmc: i2c-thermtrip node not found, emergency thermal reset disabled.
[    1.910355] Serial: 8250/16550 driver, 5 ports, IRQ sharing enabled
[    1.914976] printk: console [ttyS0] disabled
[    1.915096] 70006000.serial: ttyS0 at MMIO 0x70006000 (irq = 79, base_baud = 25500000) is a Tegra
[    2.754904] printk: console [ttyS0] enabled
[    2.760622] SuperH (H)SCI(F) driver initialized
[    2.766427] msm_serial: driver initialized
[    2.770818] STMicroelectronics ASC driver initialized
[    2.777092] 70006200.serial: ttyTHS1 at MMIO 0x70006200 (irq = 80, base_baud = 0) is a TEGRA_UART
[    2.787103] STM32 USART driver initialized
[    2.794336] tegra-host1x 50000000.host1x: Adding to iommu group 0
[    2.816556] tegra-gr2d 54140000.gr2d: Adding to iommu group 1
[    2.822957] tegra-gr3d 54180000.gr3d: Adding to iommu group 1
[    2.828804] Failed to attached device 54180000.gr3d to IOMMU_mapping
[    2.849854] brd: module loaded
[    2.862744] loop: module loaded
[    2.879727] spi-nor spi0.1: w25q32 (4096 Kbytes)
[    2.888564] libphy: Fixed MDIO Bus: probed
[    2.894913] CAN device driver interface
[    2.900140] bgmac_bcma: Broadcom 47xx GBit MAC driver loaded
[    2.907419] e1000e: Intel(R) PRO/1000 Network Driver
[    2.912391] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    2.918416] igb: Intel(R) Gigabit Ethernet Network Driver
[    2.923821] igb: Copyright (c) 2007-2014 Intel Corporation.
[    2.935677] pegasus: v0.9.3 (2013/04/25), Pegasus/Pegasus II USB Ethernet driver
[    2.943177] usbcore: registered new interface driver pegasus
[    2.948939] usbcore: registered new interface driver asix
[    2.954390] usbcore: registered new interface driver ax88179_178a
[    2.960557] usbcore: registered new interface driver cdc_ether
[    2.966462] usbcore: registered new interface driver smsc75xx
[    2.972294] usbcore: registered new interface driver smsc95xx
[    2.978123] usbcore: registered new interface driver net1080
[    2.983830] usbcore: registered new interface driver cdc_subset
[    2.989818] usbcore: registered new interface driver zaurus
[    2.995458] usbcore: registered new interface driver cdc_ncm
[    3.004756] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    3.011321] ehci-pci: EHCI PCI platform driver
[    3.015822] ehci-platform: EHCI generic platform driver
[    3.021360] ehci-orion: EHCI orion driver
[    3.025597] SPEAr-ehci: EHCI SPEAr driver
[    3.029837] ehci-st: EHCI STMicroelectronics driver
[    3.034915] ehci-atmel: EHCI Atmel driver
[    3.039144] tegra-ehci: Tegra EHCI driver
[    3.043560] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    3.049793] ohci-pci: OHCI PCI platform driver
[    3.054299] ohci-platform: OHCI generic platform driver
[    3.059816] SPEAr-ohci: OHCI SPEAr driver
[    3.064024] ohci-st: OHCI STMicroelectronics driver
[    3.069156] ohci-atmel: OHCI Atmel driver
[    3.074038] usbcore: registered new interface driver usb-storage
[    3.082350] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.093836] tegra_rtc 7000e000.rtc: registered as rtc1
[    3.099034] tegra_rtc 7000e000.rtc: Tegra internal Real Time Clock
[    3.106066] i2c /dev entries driver
[    3.123870] i2c i2c-2: Added multiplexed i2c bus 5
[    3.129052] i2c i2c-2: Added multiplexed i2c bus 6
[    3.134163] i2c i2c-2: Added multiplexed i2c bus 7
[    3.139303] i2c i2c-2: Added multiplexed i2c bus 8
[    3.144114] pca954x 2-0070: registered 4 multiplexed busses for I2C switch pca9546
[    3.166204] sdhci: Secure Digital Host Controller Interface driver
[    3.172440] sdhci: Copyright(c) Pierre Ossman
[    3.179632] Synopsys Designware Multimedia Card Interface Driver
[    3.187764] sdhci-pltfm: SDHCI platform and OF driver helper
[    3.195007] sdhci-tegra 78000000.mmc: Got CD GPIO
[    3.199821] sdhci-tegra 78000000.mmc: Got WP GPIO
[    3.204763] ledtrig-cpu: registered to indicate activity on CPUs
[    3.205820] mmc1: Invalid maximum block size, assuming 512 bytes
[    3.210978] mmc2: Invalid maximum block size, assuming 512 bytes
[    3.211933] usbcore: registered new interface driver usbhid
[    3.211942] usbhid: USB HID core driver
[    3.215632] tegra30-emc 7000f400.memory-controller: device-tree doesn't have memory timings
[    3.217075] mmc0: Invalid maximum block size, assuming 512 bytes
[    3.217936] tegra30-emc 7000f400.memory-controller: OPP HW ver. 0x4, current clock rate 800 MHz
[    3.219455] isl29028 2-0044: No cache defaults, reading back from HW
[    3.229370] NET: Registered protocol family 10
[    3.251470] mmc1: SDHCI controller on 78000400.mmc [78000400.mmc] using ADMA
[    3.256629] Segment Routing with IPv6
[    3.257399] mmc2: SDHCI controller on 78000600.mmc [78000600.mmc] using ADMA
[    3.267078] mmc0: SDHCI controller on 78000000.mmc [78000000.mmc] using ADMA
[    3.273455] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    3.297772] NET: Registered protocol family 17
[    3.302243] can: controller area network core
[    3.306682] NET: Registered protocol family 29
[    3.311181] can: raw protocol
[    3.314159] can: broadcast manager protocol
[    3.318373] can: netlink gateway - max_hops=1
[    3.323260] Key type dns_resolver registered
[    3.330006] tegra20-cpufreq tegra20-cpufreq: hardware version 0x4 0x4
[    3.337300] ThumbEE CPU extension supported.
[    3.338438] mmc2: new high speed MMC card at address 0001
[    3.341589] Registering SWP/SWPB emulation handler
[    3.341884] Loading compiled-in X.509 certificates
[    3.347785] mmcblk2: mmc2:0001 SEM16G 14.8 GiB 
[    3.348175] mmc0: new high speed SDHC card at address e624
[    3.348868] mmcblk0: mmc0:e624 SD08G 7.40 GiB 
[    3.372111] mmcblk2boot0: mmc2:0001 SEM16G partition 1 1.00 MiB
[    3.378828] mmcblk2boot1: mmc2:0001 SEM16G partition 2 1.00 MiB
[    3.385279] mmcblk2rpmb: mmc2:0001 SEM16G partition 3 128 KiB, chardev (235:0)
[    3.394231] tegra-dc 54200000.dc: Adding to iommu group 1
[    3.399742] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.402479] Alternate GPT is invalid, using primary GPT.
[    3.411326]  mmcblk2: p1 p2 p3 p4 p5 p6 p7 p8 p9
[    3.414447] tegra-dc 54200000.dc: failed to probe RGB output: -517
[    3.424816] tegra-dc 54240000.dc: Adding to iommu group 1
[    3.430327] Failed to attached device 54240000.dc to IOMMU_mapping
[    3.449126] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.469303] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.483979] tegra-dc 54200000.dc: failed to probe RGB output: -517
[    3.494351] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.514103] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.526973] tegra-dc 54200000.dc: failed to probe RGB output: -517
[    3.534641] irq: no irq domain found for tps65911@2d !
[    3.539912] gpio-keys gpio-keys: Found button without gpio or irq
[    3.546032] gpio-keys: probe of gpio-keys failed with error -22
[   46.599511] modem_3v3: disabling
[  111.879873] VFS: Unable to mount root fs via NFS.
[  111.884649] devtmpfs: mounted
[  111.890563] Freeing unused kernel memory: 2048K
[  111.918616] Run /sbin/init as init process
[  111.922735]   with arguments:
[  111.925706]     /sbin/init
[  111.928450]     earlyprintk
[  111.931256]     netdevwait
[  111.933971]   with environment:
[  111.937142]     HOME=/
[  111.939512]     TERM=linux
[  111.942475] Run /etc/init as init process
[  111.946498]   with arguments:
[  111.949491]     /etc/init
[  111.952120]     earlyprintk
[  111.954917]     netdevwait
[  111.957658]   with environment:
[  111.960806]     HOME=/
[  111.963171]     TERM=linux
[  111.965999] Run /bin/init as init process
[  111.970048]   with arguments:
[  111.973025]     /bin/init
[  111.975654]     earlyprintk
[  111.978484]     netdevwait
[  111.981200]   with environment:
[  111.984342]     HOME=/
[  111.986704]     TERM=linux
[  111.989557] Run /bin/sh as init process
[  111.993403]   with arguments:
[  111.996374]     /bin/sh
[  111.998856]     earlyprintk
[  112.001658]     netdevwait
[  112.004369]   with environment:
[  112.007549]     HOME=/
[  112.009916]     TERM=linux
[  112.012739] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[  112.026921] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  112.035634] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  112.041913] [<c03114f8>] (unwind_backtrace) from [<c030bcec>] (show_stack+0x10/0x14)
[  112.049699] [<c030bcec>] (show_stack) from [<c0fc129c>] (dump_stack+0xc8/0xdc)
[  112.056951] [<c0fc129c>] (dump_stack) from [<c0fbfb60>] (panic+0x108/0x320)
[  112.063932] [<c0fbfb60>] (panic) from [<c0fc7540>] (kernel_init+0x108/0x118)
[  112.071001] [<c0fc7540>] (kernel_init) from [<c03001b0>] (ret_from_fork+0x14/0x24)
[  112.078589] Exception stack(0xc20edfb0 to 0xc20edff8)
[  112.083650] dfa0:                                     00000000 00000000 00000000 00000000
[  112.091837] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  112.100022] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  112.106653] CPU0: stopping
[  112.109375] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  112.118085] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  112.124356] [<c03114f8>] (unwind_backtrace) from [<c030bcec>] (show_stack+0x10/0x14)
[  112.132127] [<c030bcec>] (show_stack) from [<c0fc129c>] (dump_stack+0xc8/0xdc)
[  112.139369] [<c0fc129c>] (dump_stack) from [<c030ef48>] (do_handle_IPI+0x320/0x33c)
[  112.147049] [<c030ef48>] (do_handle_IPI) from [<c030ef7c>] (ipi_handler+0x18/0x20)
[  112.154638] [<c030ef7c>] (ipi_handler) from [<c03a7b4c>] (handle_percpu_devid_irq+0x8c/0x21c)
[  112.163186] [<c03a7b4c>] (handle_percpu_devid_irq) from [<c03a16a8>] (generic_handle_irq+0x34/0x44)
[  112.172261] [<c03a16a8>] (generic_handle_irq) from [<c03a1d64>] (__handle_domain_irq+0x5c/0xb4)
[  112.180981] [<c03a1d64>] (__handle_domain_irq) from [<c0768968>] (gic_handle_irq+0x94/0xb4)
[  112.189356] [<c0768968>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
[  112.196855] Exception stack(0xc1801ea0 to 0xc1801ee8)
[  112.201917] 1ea0: 00000000 c19e8850 2e03d000 ef7927c0 00000000 c19e8850 00000000 00000000
[  112.210105] 1ec0: ef791830 c1a13560 1a13add8 0000001a fffffff6 c1801ef0 c0d4b35c c0d4b42c
[  112.218286] 1ee0: 60000113 ffffffff
[  112.221779] [<c0300b8c>] (__irq_svc) from [<c0d4b42c>] (cpuidle_enter_state+0x1c4/0x4ac)
[  112.229907] [<c0d4b42c>] (cpuidle_enter_state) from [<c0d4dd58>] (cpuidle_enter_state_coupled+0x144/0x3c8)
[  112.239586] [<c0d4dd58>] (cpuidle_enter_state_coupled) from [<c0d4b778>] (cpuidle_enter+0x50/0x54)
[  112.248560] [<c0d4b778>] (cpuidle_enter) from [<c03793c0>] (do_idle+0x204/0x280)
[  112.255980] [<c03793c0>] (do_idle) from [<c0379748>] (cpu_startup_entry+0x18/0x1c)
[  112.263568] [<c0379748>] (cpu_startup_entry) from [<c1600e54>] (start_kernel+0x514/0x558)
[  112.271770] CPU3: stopping
[  112.274494] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  112.283204] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  112.289476] [<c03114f8>] (unwind_backtrace) from [<c030bcec>] (show_stack+0x10/0x14)
[  112.297246] [<c030bcec>] (show_stack) from [<c0fc129c>] (dump_stack+0xc8/0xdc)
[  112.304489] [<c0fc129c>] (dump_stack) from [<c030ef48>] (do_handle_IPI+0x320/0x33c)
[  112.312166] [<c030ef48>] (do_handle_IPI) from [<c030ef7c>] (ipi_handler+0x18/0x20)
[  112.319755] [<c030ef7c>] (ipi_handler) from [<c03a7b4c>] (handle_percpu_devid_irq+0x8c/0x21c)
[  112.328302] [<c03a7b4c>] (handle_percpu_devid_irq) from [<c03a16a8>] (generic_handle_irq+0x34/0x44)
[  112.337369] [<c03a16a8>] (generic_handle_irq) from [<c03a1d64>] (__handle_domain_irq+0x5c/0xb4)
[  112.346089] [<c03a1d64>] (__handle_domain_irq) from [<c0768968>] (gic_handle_irq+0x94/0xb4)
[  112.354460] [<c0768968>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
[  112.361957] Exception stack(0xc211bee0 to 0xc211bf28)
[  112.367020] bee0: 00000000 c19e8850 2e079000 ef7ce7c0 00000000 c19e8850 00000000 00000000
[  112.375208] bf00: ef7cd830 c1a13560 1a13add8 0000001a fffffff6 c211bf30 c0d4b35c c0d4b42c
[  112.383390] bf20: 60000113 ffffffff
[  112.386882] [<c0300b8c>] (__irq_svc) from [<c0d4b42c>] (cpuidle_enter_state+0x1c4/0x4ac)
[  112.394999] [<c0d4b42c>] (cpuidle_enter_state) from [<c0d4dd58>] (cpuidle_enter_state_coupled+0x144/0x3c8)
[  112.404675] [<c0d4dd58>] (cpuidle_enter_state_coupled) from [<c0d4b778>] (cpuidle_enter+0x50/0x54)
[  112.413650] [<c0d4b778>] (cpuidle_enter) from [<c03793c0>] (do_idle+0x204/0x280)
[  112.421064] [<c03793c0>] (do_idle) from [<c0379748>] (cpu_startup_entry+0x18/0x1c)
[  112.428651] [<c0379748>] (cpu_startup_entry) from [<80301750>] (0x80301750)
[  112.435626] CPU2: stopping
[  112.438350] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  112.447062] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  112.453334] [<c03114f8>] (unwind_backtrace) from [<c030bcec>] (show_stack+0x10/0x14)
[  112.461107] [<c030bcec>] (show_stack) from [<c0fc129c>] (dump_stack+0xc8/0xdc)
[  112.468350] [<c0fc129c>] (dump_stack) from [<c030ef48>] (do_handle_IPI+0x320/0x33c)
[  112.476026] [<c030ef48>] (do_handle_IPI) from [<c030ef7c>] (ipi_handler+0x18/0x20)
[  112.483614] [<c030ef7c>] (ipi_handler) from [<c03a7b4c>] (handle_percpu_devid_irq+0x8c/0x21c)
[  112.492159] [<c03a7b4c>] (handle_percpu_devid_irq) from [<c03a16a8>] (generic_handle_irq+0x34/0x44)
[  112.501227] [<c03a16a8>] (generic_handle_irq) from [<c03a1d64>] (__handle_domain_irq+0x5c/0xb4)
[  112.509946] [<c03a1d64>] (__handle_domain_irq) from [<c0768968>] (gic_handle_irq+0x94/0xb4)
[  112.518317] [<c0768968>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
[  112.525816] Exception stack(0xc2113ee0 to 0xc2113f28)
[  112.530880] 3ee0: 00000000 c19e8850 2e065000 ef7ba7c0 00000000 c19e8850 00000000 00000000
[  112.539067] 3f00: ef7b9830 c1a13560 1a13add8 0000001a fffffff6 c2113f30 c0d4b35c c0d4b42c
[  112.547249] 3f20: 60000113 ffffffff
[  112.550741] [<c0300b8c>] (__irq_svc) from [<c0d4b42c>] (cpuidle_enter_state+0x1c4/0x4ac)
[  112.558857] [<c0d4b42c>] (cpuidle_enter_state) from [<c0d4dd58>] (cpuidle_enter_state_coupled+0x144/0x3c8)
[  112.568533] [<c0d4dd58>] (cpuidle_enter_state_coupled) from [<c0d4b778>] (cpuidle_enter+0x50/0x54)
[  112.577506] [<c0d4b778>] (cpuidle_enter) from [<c03793c0>] (do_idle+0x204/0x280)
[  112.584921] [<c03793c0>] (do_idle) from [<c0379748>] (cpu_startup_entry+0x18/0x1c)
[  112.592510] [<c0379748>] (cpu_startup_entry) from [<80301750>] (0x80301750)
[  112.599498] ---[ end Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. ]---

U-Boot SPL 2019.07-g2e30fa4be2 (Jan 12 2021 - 10:29:58 -0800) Trying to boot from RAM

U-Boot 2019.07-g2e30fa4be2 (Jan 12 2021 - 10:29:58 -0800)

TEGRA30

Model: NVIDIA Cardhu
Board: NVIDIA Cardhu
DRAM:  1 GiB
iBC:   sdhci@78000000: 1, sdhci@78000600: 0

Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  2  1  0 

switch to partitions #0, OK
mmc1 is current device
** No partition table - mmc 1 **
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
413 bytes read in 20 ms (19.5 KiB/s)
Cardhu NFS boot options
1: primary kernel
Enter choice: 1: primary kernel
Retrieving file: /boot/zImage
9703936 bytes read in 350 ms (26.4 MiB/s)
append: console=ttyS0,115200n8 console=tty1 earlycon earlyprintk ignore_loglevel root=/dev/nfs rw netdevwait ip=192.168.99.2:192.168.99.1:192.168.99.1:255.255.255.0::eth0:off nfsroot=192.168.99.1:/home/ausvrl2777/nfsroot,tcp rootwait Retrieving file: /boot/tegra30-cardhu-a04.dtb
44584 bytes read in 20 ms (2.1 MiB/s)
## Flattened Device Tree blob at 83000000
Booting using the fdt blob at 0x83000000
Using Device Tree in place at 83000000, end 8300de27

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.11.0-rc3-next-20210112-gdf869cab4b35 (buildbrain@mobile-u64-5252-d8000) (arm-linux-gnueabihf-gcc (Linaro GCC 7.3-2018.05) 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701], GNU ld (Linaro_Binutils-2018.05) 2.28.2.20170706) #1 SMP Tue Jan 12 10:30:14 PST 2021
[    0.000000] CPU: ARMv7 Processor [412fc099] revision 9 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: NVIDIA Tegra30 Cardhu A04 (A05, A06, A07) evaluation board
[    0.000000] earlycon: uart0 at MMIO 0x70006000 (options '115200n8')
[    0.000000] printk: bootconsole [uart0] enabled
[    0.000000] printk: debug: ignoring loglevel setting.
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 64 MiB at 0xbc000000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x00000000b0000000-0x00000000bfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] On node 0 totalpages: 262144
[    0.000000]   DMA zone: 1536 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 196608 pages, LIFO batch:63
[    0.000000]   HighMem zone: 65536 pages, LIFO batch:15
[    0.000000] percpu: Embedded 20 pages/cpu s49996 r8192 d23732 u81920
[    0.000000] pcpu-alloc: s49996 r8192 d23732 u81920 alloc=20*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 260608
[    0.000000] Kernel command line: console=ttyS0,115200n8 console=tty1 earlycon earlyprintk ignore_loglevel root=/dev/nfs rw netdevwait ip=192.168.99.2:192.168.99.1:192.168.99.1:255.255.255.0::eth0:off nfsroot=192.168.99.1:/home/ausvrl2777/nfsroot,tcp rootwait
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 949452K/1048576K available (13312K kernel code, 2183K rwdata, 5360K rodata, 2048K init, 401K bss, 33588K reserved, 65536K cma-reserved, 196608K highmem)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=4.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] /interrupt-controller@60004000: 160 interrupts forwarded to /interrupt-controller@50041000
[    0.000000] L2C: platform modifies aux control register: 0x02080000 -> 0x3e480000
[    0.000000] L2C: DT/platform modifies aux control register: 0x02080000 -> 0x3e480000
[    0.000000] L2C-310 erratum 769419 enabled
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[    0.000000] L2C-310 cache controller enabled, 8 ways, 1024 kB
[    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e480001
[    0.000000] random: get_random_bytes called from start_kernel+0x3b4/0x558 with crng_init=0
[    0.000002] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[    0.017221] clocksource: timer_us: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[    0.027139] Switching to timer-based delay loop, resolution 1000ns
[    0.034461] clocksource: tegra_suspend_timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.049721] Console: colour dummy device 80x30
[    0.054346] printk: console [tty1] enabled
[    0.058597] printk: bootconsole [uart0] disabled
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.11.0-rc3-next-20210112-gdf869cab4b35 (buildbrain@mobile-u64-5252-d8000) (arm-linux-gnueabihf-gcc (Linaro GCC 7.3-2018.05) 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701], GNU ld (Linaro_Binutils-2018.05) 2.28.2.20170706) #1 SMP Tue Jan 12 10:30:14 PST 2021
[    0.000000] CPU: ARMv7 Processor [412fc099] revision 9 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: NVIDIA Tegra30 Cardhu A04 (A05, A06, A07) evaluation board
[    0.000000] earlycon: uart0 at MMIO 0x70006000 (options '115200n8')
[    0.000000] printk: bootconsole [uart0] enabled
[    0.000000] printk: debug: ignoring loglevel setting.
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 64 MiB at 0xbc000000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x00000000b0000000-0x00000000bfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] On node 0 totalpages: 262144
[    0.000000]   DMA zone: 1536 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 196608 pages, LIFO batch:63
[    0.000000]   HighMem zone: 65536 pages, LIFO batch:15
[    0.000000] percpu: Embedded 20 pages/cpu s49996 r8192 d23732 u81920
[    0.000000] pcpu-alloc: s49996 r8192 d23732 u81920 alloc=20*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 260608
[    0.000000] Kernel command line: console=ttyS0,115200n8 console=tty1 earlycon earlyprintk ignore_loglevel root=/dev/nfs rw netdevwait ip=192.168.99.2:192.168.99.1:192.168.99.1:255.255.255.0::eth0:off nfsroot=192.168.99.1:/home/ausvrl2777/nfsroot,tcp rootwait
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 949452K/1048576K available (13312K kernel code, 2183K rwdata, 5360K rodata, 2048K init, 401K bss, 33588K reserved, 65536K cma-reserved, 196608K highmem)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=4.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] /interrupt-controller@60004000: 160 interrupts forwarded to /interrupt-controller@50041000
[    0.000000] L2C: platform modifies aux control register: 0x02080000 -> 0x3e480000
[    0.000000] L2C: DT/platform modifies aux control register: 0x02080000 -> 0x3e480000
[    0.000000] L2C-310 erratum 769419 enabled
[    0.000000] L2C-310 enabling early BRESP for Cortex-A9
[    0.000000] L2C-310 full line of zeros enabled for Cortex-A9
[    0.000000] L2C-310 dynamic clock gating enabled, standby mode enabled
[    0.000000] L2C-310 cache controller enabled, 8 ways, 1024 kB
[    0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e480001
[    0.000000] random: get_random_bytes called from start_kernel+0x3b4/0x558 with crng_init=0
[    0.000002] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns
[    0.017221] clocksource: timer_us: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[    0.027139] Switching to timer-based delay loop, resolution 1000ns
[    0.034461] clocksource: tegra_suspend_timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.049721] Console: colour dummy device 80x30
[    0.054346] printk: console [tty1] enabled
[    0.058597] printk: bootconsole [uart0] disabled
[    0.063431] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=10000)
[    0.063468] pid_max: default: 32768 minimum: 301
[    0.063695] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[    0.063736] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
[    0.064536] CPU: Testing write buffer coherency: ok
[    0.064599] CPU0: Spectre v2: using BPIALL workaround
[    0.064932] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.065570] Setting up static identity map for 0x80300000 - 0x803000ac
[    0.067705] rcu: Hierarchical SRCU implementation.
[    0.071463] Tegra Revision: A02 SKU: 129 CPU Process: 2 SoC Process: 0
[    0.072385] EFI services will not be available.
[    0.072646] smp: Bringing up secondary CPUs ...
[    0.073934] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.073953] CPU1: Spectre v2: using BPIALL workaround
[    0.083943] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[    0.083961] CPU2: Spectre v2: using BPIALL workaround
[    0.093917] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[    0.093936] CPU3: Spectre v2: using BPIALL workaround
[    0.094093] smp: Brought up 1 node, 4 CPUs
[    0.094123] SMP: Total of 4 processors activated (8.00 BogoMIPS).
[    0.094148] CPU: All CPU(s) started in SVC mode.
[    0.094802] devtmpfs: initialized
[    0.105814] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[    0.106107] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.106169] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.110634] pinctrl core: initialized pinctrl subsystem
[    0.113008] DMI not present or invalid.
[    0.113460] NET: Registered protocol family 16
[    0.117776] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.120851] thermal_sys: Registered thermal governor 'step_wise'
[    0.121095] cpuidle: using governor menu
[    0.150883] No ATAGs?
[    0.151082] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[    0.151123] hw-breakpoint: maximum watchpoint size is 4 bytes.
[    0.161726] Serial: AMBA PL011 UART driver
[    0.162815] tegra-mc 7000f000.memory-controller: no memory timings for RAM code 0 registered
[    0.190230] iommu: Default domain type: Translated 
[    0.190538] vgaarb: loaded
[    0.191496] SCSI subsystem initialized
[    0.191724] libata version 3.00 loaded.
[    0.192041] usbcore: registered new interface driver usbfs
[    0.192132] usbcore: registered new interface driver hub
[    0.192204] usbcore: registered new device driver usb
[    0.193922] pps_core: LinuxPPS API ver. 1 registered
[    0.193954] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.194004] PTP clock support registered
[    0.197397] clocksource: Switched to clocksource timer_us
[    0.266869] NET: Registered protocol family 2
[    1.347379] random: fast init done
[    1.780308] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[    1.780380] TCP established hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    1.780506] TCP bind hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    1.780695] TCP: Hash tables configured (established 8192 bind 8192)
[    1.780865] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[    1.780953] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[    1.781217] NET: Registered protocol family 1
[    1.781862] RPC: Registered named UNIX socket transport module.
[    1.781898] RPC: Registered udp transport module.
[    1.781921] RPC: Registered tcp transport module.
[    1.781942] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.781970] PCI: CLS 0 bytes, default 64
[    1.783646] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[    1.785073] Initialise system trusted keyrings
[    1.785304] workingset: timestamp_bits=30 max_order=18 bucket_order=0
[    1.792914] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.793763] NFS: Registering the id_resolver key type
[    1.793822] Key type id_resolver registered
[    1.793847] Key type id_legacy registered
[    1.793985] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.794048] ntfs: driver 2.1.32 [Flags: R/O].
[    1.794555] Key type asymmetric registered
[    1.794585] Asymmetric key parser 'x509' registered
[    1.794672] bounce: pool size: 64 pages
[    1.794738] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    1.794773] io scheduler mq-deadline registered
[    1.794797] io scheduler kyber registered
[    1.830508] tegra-apbdma 6000a000.dma: Tegra20 APB DMA driver registered 32 channels
[    1.836744] tegra-pmc 7000e400.pmc: i2c-thermtrip node not found, emergency thermal reset disabled.
[    1.905419] Serial: 8250/16550 driver, 5 ports, IRQ sharing enabled
[    1.910106] printk: console [ttyS0] disabled
[    1.910225] 70006000.serial: ttyS0 at MMIO 0x70006000 (irq = 79, base_baud = 25500000) is a Tegra
[    2.750136] printk: console [ttyS0] enabled
[    2.755816] SuperH (H)SCI(F) driver initialized
[    2.761723] msm_serial: driver initialized
[    2.766053] STMicroelectronics ASC driver initialized
[    2.772337] 70006200.serial: ttyTHS1 at MMIO 0x70006200 (irq = 80, base_baud = 0) is a TEGRA_UART
[    2.782331] STM32 USART driver initialized
[    2.789553] tegra-host1x 50000000.host1x: Adding to iommu group 0
[    2.811767] tegra-gr2d 54140000.gr2d: Adding to iommu group 1
[    2.818147] tegra-gr3d 54180000.gr3d: Adding to iommu group 1
[    2.823942] Failed to attached device 54180000.gr3d to IOMMU_mapping
[    2.844969] brd: module loaded
[    2.857913] loop: module loaded
[    2.874830] spi-nor spi0.1: w25q32 (4096 Kbytes)
[    2.883651] libphy: Fixed MDIO Bus: probed
[    2.890030] CAN device driver interface
[    2.895232] bgmac_bcma: Broadcom 47xx GBit MAC driver loaded
[    2.902489] e1000e: Intel(R) PRO/1000 Network Driver
[    2.907485] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    2.913498] igb: Intel(R) Gigabit Ethernet Network Driver
[    2.918927] igb: Copyright (c) 2007-2014 Intel Corporation.
[    2.930757] pegasus: v0.9.3 (2013/04/25), Pegasus/Pegasus II USB Ethernet driver
[    2.938248] usbcore: registered new interface driver pegasus
[    2.943976] usbcore: registered new interface driver asix
[    2.949459] usbcore: registered new interface driver ax88179_178a
[    2.955603] usbcore: registered new interface driver cdc_ether
[    2.961538] usbcore: registered new interface driver smsc75xx
[    2.967347] usbcore: registered new interface driver smsc95xx
[    2.973166] usbcore: registered new interface driver net1080
[    2.978899] usbcore: registered new interface driver cdc_subset
[    2.984862] usbcore: registered new interface driver zaurus
[    2.990535] usbcore: registered new interface driver cdc_ncm
[    2.999819] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    3.006357] ehci-pci: EHCI PCI platform driver
[    3.010879] ehci-platform: EHCI generic platform driver
[    3.016394] ehci-orion: EHCI orion driver
[    3.020658] SPEAr-ehci: EHCI SPEAr driver
[    3.024874] ehci-st: EHCI STMicroelectronics driver
[    3.029982] ehci-atmel: EHCI Atmel driver
[    3.034182] tegra-ehci: Tegra EHCI driver
[    3.038627] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    3.044832] ohci-pci: OHCI PCI platform driver
[    3.049362] ohci-platform: OHCI generic platform driver
[    3.054853] SPEAr-ohci: OHCI SPEAr driver
[    3.059090] ohci-st: OHCI STMicroelectronics driver
[    3.064166] ohci-atmel: OHCI Atmel driver
[    3.069070] usbcore: registered new interface driver usb-storage
[    3.077303] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.088913] tegra_rtc 7000e000.rtc: registered as rtc1
[    3.094073] tegra_rtc 7000e000.rtc: Tegra internal Real Time Clock
[    3.101144] i2c /dev entries driver
[    3.118967] i2c i2c-2: Added multiplexed i2c bus 5
[    3.124089] i2c i2c-2: Added multiplexed i2c bus 6
[    3.129260] i2c i2c-2: Added multiplexed i2c bus 7
[    3.134366] i2c i2c-2: Added multiplexed i2c bus 8
[    3.139207] pca954x 2-0070: registered 4 multiplexed busses for I2C switch pca9546
[    3.161392] sdhci: Secure Digital Host Controller Interface driver
[    3.167619] sdhci: Copyright(c) Pierre Ossman
[    3.174720] Synopsys Designware Multimedia Card Interface Driver
[    3.182829] sdhci-pltfm: SDHCI platform and OF driver helper
[    3.190081] sdhci-tegra 78000000.mmc: Got CD GPIO
[    3.194852] sdhci-tegra 78000000.mmc: Got WP GPIO
[    3.199773] ledtrig-cpu: registered to indicate activity on CPUs
[    3.201620] mmc2: Invalid maximum block size, assuming 512 bytes
[    3.201668] mmc1: Invalid maximum block size, assuming 512 bytes
[    3.206920] usbcore: registered new interface driver usbhid
[    3.207883] mmc0: Invalid maximum block size, assuming 512 bytes
[    3.229485] usbhid: USB HID core driver
[    3.236132] mmc1: SDHCI controller on 78000400.mmc [78000400.mmc] using ADMA
[    3.237197] tegra30-emc 7000f400.memory-controller: device-tree doesn't have memory timings
[    3.243489] mmc0: SDHCI controller on 78000000.mmc [78000000.mmc] using ADMA
[    3.246272] mmc2: SDHCI controller on 78000600.mmc [78000600.mmc] using ADMA
[    3.253838] tegra30-emc 7000f400.memory-controller: OPP HW ver. 0x4, current clock rate 800 MHz
[    3.276001] isl29028 2-0044: No cache defaults, reading back from HW
[    3.289228] NET: Registered protocol family 10
[    3.294948] Segment Routing with IPv6
[    3.298744] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    3.305369] NET: Registered protocol family 17
[    3.309872] can: controller area network core
[    3.314307] NET: Registered protocol family 29
[    3.318809] can: raw protocol
[    3.321785] can: broadcast manager protocol
[    3.325978] can: netlink gateway - max_hops=1
[    3.330833] Key type dns_resolver registered
[    3.337611] tegra20-cpufreq tegra20-cpufreq: hardware version 0x4 0x4
[    3.344844] ThumbEE CPU extension supported.
[    3.349169] Registering SWP/SWPB emulation handler
[    3.354461] Loading compiled-in X.509 certificates
[    3.402314] tegra-dc 54200000.dc: Adding to iommu group 1
[    3.407827] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.421548] tegra-dc 54200000.dc: failed to probe RGB output: -517
[    3.430054] tegra-dc 54240000.dc: Adding to iommu group 1
[    3.435520] Failed to attached device 54240000.dc to IOMMU_mapping
[    3.445450] mmc0: new high speed SDHC card at address e624
[    3.448233] mmc2: new high speed MMC card at address 0001
[    3.451780] mmcblk0: mmc0:e624 SD08G 7.40 GiB 
[    3.453853] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.457495] mmcblk2: mmc2:0001 SEM16G 14.8 GiB 
[    3.469904] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.471325] mmcblk2boot0: mmc2:0001 SEM16G partition 1 1.00 MiB
[    3.483446] mmcblk2boot1: mmc2:0001 SEM16G partition 2 1.00 MiB
[    3.485372] tegra-dc 54200000.dc: failed to probe RGB output: -517
[    3.489656] mmcblk2rpmb: mmc2:0001 SEM16G partition 3 128 KiB, chardev (235:0)
[    3.500550] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.523574] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.532241] Alternate GPT is invalid, using primary GPT.
[    3.537637]  mmcblk2: p1 p2 p3 p4 p5 p6 p7 p8 p9
[    3.538085] tegra-dc 54200000.dc: failed to probe RGB output: -517
[    3.553525] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.573708] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.587497] tegra-dc 54200000.dc: failed to probe RGB output: -517
[    3.598013] tegra-usb 7d008000.usb: failed to get PHY: -517
[    3.617897] Failed to attached device 54200000.dc to IOMMU_mapping
[    3.631509] tegra-dc 54200000.dc: failed to probe RGB output: -517
[    3.639351] irq: no irq domain found for tps65911@2d !
[    3.644605] gpio-keys gpio-keys: Found button without gpio or irq
[    3.650766] gpio-keys: probe of gpio-keys failed with error -22
[   46.599975] modem_3v3: disabling
[  111.880354] VFS: Unable to mount root fs via NFS.
[  111.885126] devtmpfs: mounted
[  111.891018] Freeing unused kernel memory: 2048K
[  111.938807] Run /sbin/init as init process
[  111.942927]   with arguments:
[  111.945896]     /sbin/init
[  111.948641]     earlyprintk
[  111.951446]     netdevwait
[  111.954161]   with environment:
[  111.957304]     HOME=/
[  111.959689]     TERM=linux
[  111.962650] Run /etc/init as init process
[  111.966674]   with arguments:
[  111.969671]     /etc/init
[  111.972300]     earlyprintk
[  111.975098]     netdevwait
[  111.977842]   with environment:
[  111.980990]     HOME=/
[  111.983354]     TERM=linux
[  111.986184] Run /bin/init as init process
[  111.990241]   with arguments:
[  111.993217]     /bin/init
[  111.995841]     earlyprintk
[  111.998673]     netdevwait
[  112.001387]   with environment:
[  112.004529]     HOME=/
[  112.006891]     TERM=linux
[  112.009740] Run /bin/sh as init process
[  112.013586]   with arguments:
[  112.016555]     /bin/sh
[  112.019036]     earlyprintk
[  112.021837]     netdevwait
[  112.024546]   with environment:
[  112.027727]     HOME=/
[  112.030093]     TERM=linux
[  112.032917] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[  112.047099] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  112.055812] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  112.062091] [<c03114f8>] (unwind_backtrace) from [<c030bcec>] (show_stack+0x10/0x14)
[  112.069880] [<c030bcec>] (show_stack) from [<c0fc129c>] (dump_stack+0xc8/0xdc)
[  112.077133] [<c0fc129c>] (dump_stack) from [<c0fbfb60>] (panic+0x108/0x320)
[  112.084113] [<c0fbfb60>] (panic) from [<c0fc7540>] (kernel_init+0x108/0x118)
[  112.091184] [<c0fc7540>] (kernel_init) from [<c03001b0>] (ret_from_fork+0x14/0x24)
[  112.098772] Exception stack(0xc20edfb0 to 0xc20edff8)
[  112.103832] dfa0:                                     00000000 00000000 00000000 00000000
[  112.112019] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  112.120204] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  112.126833] CPU1: stopping
[  112.129554] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  112.138265] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  112.144538] [<c03114f8>] (unwind_backtrace) from [<c030bcec>] (show_stack+0x10/0x14)
[  112.152307] [<c030bcec>] (show_stack) from [<c0fc129c>] (dump_stack+0xc8/0xdc)
[  112.159550] [<c0fc129c>] (dump_stack) from [<c030ef48>] (do_handle_IPI+0x320/0x33c)
[  112.167228] [<c030ef48>] (do_handle_IPI) from [<c030ef7c>] (ipi_handler+0x18/0x20)
[  112.174816] [<c030ef7c>] (ipi_handler) from [<c03a7b4c>] (handle_percpu_devid_irq+0x8c/0x21c)
[  112.183365] [<c03a7b4c>] (handle_percpu_devid_irq) from [<c03a16a8>] (generic_handle_irq+0x34/0x44)
[  112.192439] [<c03a16a8>] (generic_handle_irq) from [<c03a1d64>] (__handle_domain_irq+0x5c/0xb4)
[  112.201159] [<c03a1d64>] (__handle_domain_irq) from [<c0768968>] (gic_handle_irq+0x94/0xb4)
[  112.209536] [<c0768968>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
[  112.217035] Exception stack(0xc2111ee0 to 0xc2111f28)
[  112.222098] 1ee0: 00000000 c19e8850 2e051000 ef7a67c0 00000000 c19e8850 00000000 00000000
[  112.230286] 1f00: ef7a5830 c1a13560 1b479de0 0000001a fffffff6 c2111f30 c0d4b35c c0d4b42c
[  112.238467] 1f20: 60000113 ffffffff
[  112.241959] [<c0300b8c>] (__irq_svc) from [<c0d4b42c>] (cpuidle_enter_state+0x1c4/0x4ac)
[  112.250088] [<c0d4b42c>] (cpuidle_enter_state) from [<c0d4dd58>] (cpuidle_enter_state_coupled+0x144/0x3c8)
[  112.259767] [<c0d4dd58>] (cpuidle_enter_state_coupled) from [<c0d4b778>] (cpuidle_enter+0x50/0x54)
[  112.268742] [<c0d4b778>] (cpuidle_enter) from [<c03793c0>] (do_idle+0x204/0x280)
[  112.276161] [<c03793c0>] (do_idle) from [<c0379748>] (cpu_startup_entry+0x18/0x1c)
[  112.283750] [<c0379748>] (cpu_startup_entry) from [<80301750>] (0x80301750)
[  112.290727] CPU2: stopping
[  112.293451] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  112.302162] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  112.308434] [<c03114f8>] (unwind_backtrace) from [<c030bcec>] (show_stack+0x10/0x14)
[  112.316205] [<c030bcec>] (show_stack) from [<c0fc129c>] (dump_stack+0xc8/0xdc)
[  112.323448] [<c0fc129c>] (dump_stack) from [<c030ef48>] (do_handle_IPI+0x320/0x33c)
[  112.331126] [<c030ef48>] (do_handle_IPI) from [<c030ef7c>] (ipi_handler+0x18/0x20)
[  112.338715] [<c030ef7c>] (ipi_handler) from [<c03a7b4c>] (handle_percpu_devid_irq+0x8c/0x21c)
[  112.347260] [<c03a7b4c>] (handle_percpu_devid_irq) from [<c03a16a8>] (generic_handle_irq+0x34/0x44)
[  112.356328] [<c03a16a8>] (generic_handle_irq) from [<c03a1d64>] (__handle_domain_irq+0x5c/0xb4)
[  112.365048] [<c03a1d64>] (__handle_domain_irq) from [<c0768968>] (gic_handle_irq+0x94/0xb4)
[  112.373419] [<c0768968>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
[  112.380918] Exception stack(0xc2113ee0 to 0xc2113f28)
[  112.385982] 3ee0: 00000000 c19e8850 2e065000 ef7ba7c0 00000000 c19e8850 00000000 00000000
[  112.394170] 3f00: ef7b9830 c1a13560 1b479de0 0000001a fffffff6 c2113f30 c0d4b35c c0d4b42c
[  112.402352] 3f20: 60000113 ffffffff
[  112.405844] [<c0300b8c>] (__irq_svc) from [<c0d4b42c>] (cpuidle_enter_state+0x1c4/0x4ac)
[  112.413962] [<c0d4b42c>] (cpuidle_enter_state) from [<c0d4dd58>] (cpuidle_enter_state_coupled+0x144/0x3c8)
[  112.423638] [<c0d4dd58>] (cpuidle_enter_state_coupled) from [<c0d4b778>] (cpuidle_enter+0x50/0x54)
[  112.432612] [<c0d4b778>] (cpuidle_enter) from [<c03793c0>] (do_idle+0x204/0x280)
[  112.440027] [<c03793c0>] (do_idle) from [<c0379748>] (cpu_startup_entry+0x18/0x1c)
[  112.447614] [<c0379748>] (cpu_startup_entry) from [<80301750>] (0x80301750)
[  112.454589] CPU3: stopping
[  112.457312] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 5.11.0-rc3-next-20210112-gdf869cab4b35 #1
[  112.466023] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  112.472294] [<c03114f8>] (unwind_backtrace) from [<c030bcec>] (show_stack+0x10/0x14)
[  112.480065] [<c030bcec>] (show_stack) from [<c0fc129c>] (dump_stack+0xc8/0xdc)
[  112.487307] [<c0fc129c>] (dump_stack) from [<c030ef48>] (do_handle_IPI+0x320/0x33c)
[  112.494983] [<c030ef48>] (do_handle_IPI) from [<c030ef7c>] (ipi_handler+0x18/0x20)
[  112.502571] [<c030ef7c>] (ipi_handler) from [<c03a7b4c>] (handle_percpu_devid_irq+0x8c/0x21c)
[  112.511115] [<c03a7b4c>] (handle_percpu_devid_irq) from [<c03a16a8>] (generic_handle_irq+0x34/0x44)
[  112.520182] [<c03a16a8>] (generic_handle_irq) from [<c03a1d64>] (__handle_domain_irq+0x5c/0xb4)
[  112.528901] [<c03a1d64>] (__handle_domain_irq) from [<c0768968>] (gic_handle_irq+0x94/0xb4)
[  112.537272] [<c0768968>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
[  112.544770] Exception stack(0xc211bee0 to 0xc211bf28)
[  112.549831] bee0: 00000000 c19e8850 2e079000 ef7ce7c0 00000000 c19e8850 00000000 00000000
[  112.558020] bf00: ef7cd830 c1a13560 1b479de0 0000001a fffffff6 c211bf30 c0d4b35c c0d4b42c
[  112.566201] bf20: 60000113 ffffffff
[  112.569693] [<c0300b8c>] (__irq_svc) from [<c0d4b42c>] (cpuidle_enter_state+0x1c4/0x4ac)
[  112.577808] [<c0d4b42c>] (cpuidle_enter_state) from [<c0d4dd58>] (cpuidle_enter_state_coupled+0x144/0x3c8)
[  112.587483] [<c0d4dd58>] (cpuidle_enter_state_coupled) from [<c0d4b778>] (cpuidle_enter+0x50/0x54)
[  112.596457] [<c0d4b778>] (cpuidle_enter) from [<c03793c0>] (do_idle+0x204/0x280)
[  112.603872] [<c03793c0>] (do_idle) from [<c0379748>] (cpu_startup_entry+0x18/0x1c)
[  112.611459] [<c0379748>] (cpu_startup_entry) from [<80301750>] (0x80301750)
[  112.618447] ---[ end Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. ]---
#69 lore raw
Quoted 57 lines · Jon Hunter
Quoted 54 lines · Saravana Kannan
Quoted 37 lines · Jon Hunter

On 13/01/2021 21:29, Saravana Kannan wrote:

...

Quoted 31 lines
Quoted 30 lines

I am seeing the same problem on Tegra30 Cardhu A04 where several regulators are continuously deferred and prevents the board from booting ...

[    2.518334] platform panel: probe deferral - supplier regulator@11 not ready
[    2.525503] platform regulator@1: probe deferral - supplier 4-002d not ready
[    2.533141] platform regulator@3: probe deferral - supplier regulator@101 not ready
[    2.540856] platform regulator@5: probe deferral - supplier regulator@101 not ready
[    2.548589] platform regulator@6: probe deferral - supplier regulator@101 not ready
[    2.556316] platform regulator@7: probe deferral - supplier regulator@101 not ready
[    2.564041] platform regulator@8: probe deferral - supplier regulator@101 not ready
[    2.571743] platform regulator@9: probe deferral - supplier regulator@101 not ready
[    2.579463] platform regulator@10: probe deferral - supplier regulator@101 not ready
[    2.587273] platform regulator@11: probe deferral - supplier regulator@101 not ready
[    2.595088] platform regulator@12: probe deferral - supplier regulator@104 not ready
[    2.603837] platform regulator@102: probe deferral - supplier regulator@104 not ready
[    2.611726] platform regulator@103: probe deferral - supplier regulator@104 not ready
[    2.620137] platform 3000.pcie: probe deferral - supplier regulator@5 not ready
Quoted 7 lines
Quoted 3 lines

Looks like this is not the whole log? Do you see any "wait for supplier" logs? That's what all these boot issues should boil down to. And as usual, pointer to DT for this board please.

Ah yes I see ...

platform regulator@1: probe deferral - wait for supplier tps65911@2d

Do you mind sharing the full log please? It's hard to tell you anything useful with bits and pieces of logs.

Quoted 4 lines

Yes the device-tree for this board can be found here [0]. Looks like there is a circular dependency between the vddctrl_reg and vddcore_reg. This is part of coupled regulators which have a two-way linkage [1]. So this change appears to conflict with this.

Quoted 8 lines
Quoted 5 lines

fw_devlink doesn't track "regulator-coupled-with". So that's probably not it. Also, this patch series was made to handle simple cycles properly. It'll functionally disable the device links it created when it comes to probe ordering. Only two overlapping cycles might cause issues -- and even that, not all the time. So yeah, full log please.

No problem. Please find attached.

Thanks! I think you forgot to enable those logs though. Also, while you are at it, maybe enable the logs in device_link_add() too please?

-Saravana

Quoted 7 lines

Cheers
Jon

--
nvpublic