Skip to contentSkip to conversation

driver core: Add device link related sysfs files

PATCH In review Saravana Kannan 4 participants 15 messages May 21, 2020 – Jul 15, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v3 cover driver core: Add device link related sysfs files

With fw_devlink and with sync_state() callback features, there's a lot of device/device link related information that's not available in sysfs.

Exposing these details to user space can be very useful in understanding suspend/resume issues, runtime pm issues, probing issues, figuring out the modules that'd be needed for first stage init, etc. In fact, an earlier verion of this series was very helpful in debugging and validating the recent memory leak fix[1].

This series is based on driver-core-next and [1] cherry-picked on top of it.

[1] - https://lore.kernel.org/lkml/[email protected]/

v1->v2:
Patch 1/4
- New patch
Patch 2/4
- Fixed the warnings I saw before that were related to incorrect sysfs removal code when a device link is deleted.
- Fixed error handling in device_link_add()
- Split up flags into more meaningful files.
- Added status file.
Patch 3/4
- Fixed error handling that Greg pointed out before. Patch 4/4
- New patch

v2->v3:
- Dropped patch 1/4 from v2 since it was already picked up. So numbering has shifted.
Patch 1/3
- Added documentation

Saravana Kannan (3):
driver core: Expose device link details in sysfs
driver core: Add state_synced sysfs file for devices that support it driver core: Add waiting_for_supplier sysfs file for devices

 Documentation/ABI/testing/sysfs-class-devlink | 126 ++++++++++
 .../ABI/testing/sysfs-devices-consumer        |   8 +
 .../ABI/testing/sysfs-devices-state_synced    |  24 ++
 .../ABI/testing/sysfs-devices-supplier        |   8 +
 .../sysfs-devices-waiting_for_supplier        |  17 ++
 drivers/base/core.c                           | 237 +++++++++++++++++-
 drivers/base/dd.c                             |  22 ++
 include/linux/device.h                        |  58 ++---
 8 files changed, 464 insertions(+), 36 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-class-devlink
 create mode 100644 Documentation/ABI/testing/sysfs-devices-consumer
 create mode 100644 Documentation/ABI/testing/sysfs-devices-state_synced
 create mode 100644 Documentation/ABI/testing/sysfs-devices-supplier
 create mode 100644 Documentation/ABI/testing/sysfs-devices-waiting_for_supplier
Signature · 1 line
2.27.0.rc0.183.gde8f92d652-goog
#1 lore raw

PATCH v3 1/3 driver core: Expose device link details in sysfs

It's helpful to be able to look at device link details from sysfs. So, expose it in sysfs.

Say device-A is supplier of device-B. These are the additional files this patch would create:

/sys/class/devlink/device-A:device-B/

	auto_remove_on
	consumer/ -> .../device-B/
	runtime_pm
	status
	supplier/ -> .../device-A/
	sync_state_only

/sys/devices/.../device-A/

	consumer:device-B/ -> /sys/class/devlink/device-A:device-B/

/sys/devices/.../device-B/

	supplier:device-A/ -> /sys/class/devlink/device-A:device-B/

That way:
To get a list of all the device link in the system:
ls /sys/class/devlink/

To get the consumer names and links of a device:
ls -d /sys/devices/.../device-X/consumer:*

To get the supplier names and links of a device:
ls -d /sys/devices/.../device-X/supplier:*

---
 Documentation/ABI/testing/sysfs-class-devlink | 126 +++++++++++
 .../ABI/testing/sysfs-devices-consumer        |   8 +
 .../ABI/testing/sysfs-devices-supplier        |   8 +
 drivers/base/core.c                           | 211 +++++++++++++++++-
 include/linux/device.h                        |  58 ++---
 5 files changed, 375 insertions(+), 36 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-class-devlink
 create mode 100644 Documentation/ABI/testing/sysfs-devices-consumer
 create mode 100644 Documentation/ABI/testing/sysfs-devices-supplier
Documentation/ABI/testing/sysfs-class-devlink new file +126 -0
new file mode 100644
index 000000000000..3a24973abb83
@@ -0,0 +1,126 @@
+What:		/sys/class/devlink/.../
+Date:		May 2020
+Contact:	Saravana Kannan <[email protected]>
+Description:
+		Provide a place in sysfs for the device link objects in the
+		kernel at any given time.  The name of a device link directory,
+		denoted as ... above, is of the form <supplier>:<consumer>
+		where <supplier> is the supplier device name and <consumer> is
+		the consumer device name.
+
+What:		/sys/class/devlink/.../auto_remove_on
+Date:		May 2020
+Contact:	Saravana Kannan <[email protected]>
+Description:
+		This file indicates if the device link will ever be
+		automatically removed by the driver core when the consumer and
+		supplier devices themselves are still present.
+
+		This will be one of the following strings:
+
+		'consumer unbind'
+		'supplier unbind'
+		'never'
+
+		'consumer unbind' means the device link will be removed when
+		the consumer's driver is unbound from the consumer device.
+
+		'supplier unbind' means the device link will be removed when
+		the supplier's driver is unbound from the supplier device.
+
+		'never' means the device link will not be automatically removed
+		when as long as the supplier and consumer devices themselves
+		are still present.
+
+What:		/sys/class/devlink/.../consumer
+Date:		May 2020
+Contact:	Saravana Kannan <[email protected]>
+Description:
+		This file is a symlink to the consumer device's sysfs directory.
+
+What:		/sys/class/devlink/.../runtime_pm
+Date:		May 2020
+Contact:	Saravana Kannan <[email protected]>
+Description:
+		This file indicates if the device link has any impact on the
+		runtime power management behavior of the consumer and supplier
+		devices. For example: Making sure the supplier doesn't enter
+		runtime suspend while the consumer is active.
+
+		This will be one of the following strings:
+
+		'0' - Does not affect runtime power management
+		'1' - Affects runtime power management
+
+What:		/sys/class/devlink/.../status
+Date:		May 2020
+Contact:	Saravana Kannan <[email protected]>
+Description:
+		This file indicates the status of the device link. The status
+		of a device link is affected by whether the supplier and
+		consumer devices have been bound to their corresponding
+		drivers. The status of a device link also affects the binding
+		and unbinding of the supplier and consumer devices with their
+		drivers and also affects whether the software state of the
+		supplier device is synced with the hardware state of the
+		supplier device after boot up.
+		See also: sysfs-devices-state_synced.
+
+		This will be one of the following strings:
+
+		'not tracked'
+		'dormant'
+		'available'
+		'consumer probing'
+		'active'
+		'supplier unbinding'
+		'unknown'
+
+		'not tracked' means this device link does not track the status
+		and has no impact on the binding, unbinding and syncing the
+		hardware and software device state.
+
+		'dormant' means the supplier and the consumer devices have not
+		bound to their driver.
+
+		'available' means the supplier has bound to its driver and is
+		available to supply resources to the consumer device.
+
+		'consumer probing' means the consumer device is currently
+		trying to bind to its driver.
+
+		'active' means the supplier and consumer devices have both
+		bound successfully to their drivers.
+
+		'supplier unbinding' means the supplier devices is currently in
+		the process of unbinding from its driver.
+
+		'unknown' means the state of the device link is not any of the
+		above. If this is ever the value, there's a bug in the kernel.
+
+What:		/sys/class/devlink/.../supplier
+Date:		May 2020
+Contact:	Saravana Kannan <[email protected]>
+Description:
+		This file is a symlink to the supplier device's sysfs directory.
+
+What:		/sys/class/devlink/.../sync_state_only
+Date:		May 2020
+Contact:	Saravana Kannan <[email protected]>
+Description:
+		This file indicates if the device link is limited to only
+		affecting the syncing of the hardware and software state of the
+		supplier device.
+
+		This will be one of the following strings:
+
+		'0'
+		'1' - Affects runtime power management
+
+		'0' means the device link can affect other device behaviors
+		like binding/unbinding, suspend/resume, runtime power
+		management, etc.
+
+		'1' means the device link will only affect the syncing of
+		hardware and software state of the supplier device after boot
+		up and doesn't not affect other behaviors of the devices.
Documentation/ABI/testing/sysfs-devices-consumer new file +8 -0
new file mode 100644
index 000000000000..1f06d74d1c3c
@@ -0,0 +1,8 @@
+What:		/sys/devices/.../consumer:<consumer>
+Date:		May 2020
+Contact:	Saravana Kannan <[email protected]>
+Description:
+		The /sys/devices/.../consumer:<consumer> are symlinks to device
+		links where this device is the supplier. <consumer> denotes the
+		name of the consumer in that device link. There can be zero or
+		more of these symlinks for a given device.
Documentation/ABI/testing/sysfs-devices-supplier new file +8 -0
new file mode 100644
index 000000000000..a919e0db5e90
@@ -0,0 +1,8 @@
+What:		/sys/devices/.../supplier:<supplier>
+Date:		May 2020
+Contact:	Saravana Kannan <[email protected]>
+Description:
+		The /sys/devices/.../supplier:<supplier> are symlinks to device
+		links where this device is the consumer. <supplier> denotes the
+		name of the supplier in that device link. There can be zero or
+		more of these symlinks for a given device.
drivers/base/core.c +203 -8

241 lines not on this page · raw message

include/linux/device.h +30 -28

70 lines not on this page · raw message

Signature · 1 line
2.27.0.rc0.183.gde8f92d652-goog
#2 lore raw

Re: [PATCH v3 1/3] driver core: Expose device link details in sysfs

Hi,

Quoted 24 lines · Saravana Kannan

It's helpful to be able to look at device link details from sysfs. So, expose it in sysfs.

Say device-A is supplier of device-B. These are the additional files this patch would create:

/sys/class/devlink/device-A:device-B/

	auto_remove_on
	consumer/ -> .../device-B/
	runtime_pm
	status
	supplier/ -> .../device-A/
	sync_state_only

/sys/devices/.../device-A/

	consumer:device-B/ -> /sys/class/devlink/device-A:device-B/

/sys/devices/.../device-B/

	supplier:device-A/ -> /sys/class/devlink/device-A:device-B/

That way:
To get a list of all the device link in the system:
ls /sys/class/devlink/

Quoted 7 lines

To get the consumer names and links of a device:
ls -d /sys/devices/.../device-X/consumer:*

To get the supplier names and links of a device:
ls -d /sys/devices/.../device-X/supplier:*

This patch creates a lot of noise in linux-next when booting various qemu images. Example log output as well as bisect results below. Reverting the patch fixes the problem.

Guenter

---

BUG: sleeping function called from invalid context at kernel/locking/mutex.c:935

in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 13, name: kworker/0:1 2 locks held by kworker/0:1/13:
#0: c78048a4 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: process_one_work+0x1a0/0x728 #1: c78ebf18 ((work_completion)(&sdp->work)){+.+.}-{0:0}, at: process_one_work+0x1a0/0x728
Preemption disabled at:

[<c0085bb4>] srcu_invoke_callbacks+0xc0/0x158
CPU: 0 PID: 13 Comm: kworker/0:1 Not tainted 5.8.0-rc5-next-20200715 #1
Hardware name: Freescale i.MX25 (Device Tree Support)
Workqueue: rcu_gp srcu_invoke_callbacks
[<c0010fe4>] (unwind_backtrace) from [<c000e740>] (show_stack+0x10/0x18)
[<c000e740>] (show_stack) from [<c050985c>] (dump_stack+0x20/0x2c)
[<c050985c>] (dump_stack) from [<c00496e4>] (___might_sleep+0x1d0/0x2cc)
[<c00496e4>] (___might_sleep) from [<c09105e8>] (__mutex_lock+0x38/0x654)
[<c09105e8>] (__mutex_lock) from [<c0910c20>] (mutex_lock_nested+0x1c/0x28)
[<c0910c20>] (mutex_lock_nested) from [<c05b25dc>] (device_del+0x2c/0x3f0)
[<c05b25dc>] (device_del) from [<c05b2a10>] (__device_link_free_srcu+0x50/0x64)
[<c05b2a10>] (__device_link_free_srcu) from [<c0085bc4>] (srcu_invoke_callbacks+0xd0/0x158)
[<c0085bc4>] (srcu_invoke_callbacks) from [<c003ad2c>] (process_one_work+0x284/0x728)
[<c003ad2c>] (process_one_work) from [<c003b1fc>] (worker_thread+0x2c/0x52c)
[<c003b1fc>] (worker_thread) from [<c0042df8>] (kthread+0x154/0x180)
[<c0042df8>] (kthread) from [<c0008530>] (ret_from_fork+0x14/0x24)
Exception stack(0xc78ebfb0 to 0xc78ebff8)
bfa0:                                     00000000 00000000 00000000 00000000
bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
BUG: scheduling while atomic: kworker/0:1/13/0x00000201
3 locks held by kworker/0:1/13:
 #0: c78048a4 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: process_one_work+0x1a0/0x728
 #1: c78ebf18 ((work_completion)(&sdp->work)){+.+.}-{0:0}, at: process_one_work+0x1a0/0x728
 #2: c0cf5710 (gdp_mutex){+.+.}-{3:3}, at: cleanup_glue_dir.part.14+0x14/0xa0
Modules linked in:
Preemption disabled at:
[<c0085bb4>] srcu_invoke_callbacks+0xc0/0x158
CPU: 0 PID: 13 Comm: kworker/0:1 Tainted: G        W         5.8.0-rc5-next-20200715 #1
Hardware name: Freescale i.MX25 (Device Tree Support)
Workqueue: rcu_gp srcu_invoke_callbacks
[<c0010fe4>] (unwind_backtrace) from [<c000e740>] (show_stack+0x10/0x18)
[<c000e740>] (show_stack) from [<c050985c>] (dump_stack+0x20/0x2c)
[<c050985c>] (dump_stack) from [<c0049910>] (__schedule_bug+0x88/0xf0)
[<c0049910>] (__schedule_bug) from [<c090e148>] (__schedule+0x5f0/0x824)
[<c090e148>] (__schedule) from [<c090e3d0>] (schedule+0x54/0x128)
[<c090e3d0>] (schedule) from [<c090e838>] (schedule_preempt_disabled+0x14/0x28)
[<c090e838>] (schedule_preempt_disabled) from [<c0910744>] (__mutex_lock+0x194/0x654)
[<c0910744>] (__mutex_lock) from [<c0910c20>] (mutex_lock_nested+0x1c/0x28)
[<c0910c20>] (mutex_lock_nested) from [<c05b1d5c>] (cleanup_glue_dir.part.14+0x14/0xa0)
[<c05b1d5c>] (cleanup_glue_dir.part.14) from [<c05b296c>] (device_del+0x3bc/0x3f0)
[<c05b296c>] (device_del) from [<c05b2a10>] (__device_link_free_srcu+0x50/0x64)
[<c05b2a10>] (__device_link_free_srcu) from [<c0085bc4>] (srcu_invoke_callbacks+0xd0/0x158)
[<c0085bc4>] (srcu_invoke_callbacks) from [<c003ad2c>] (process_one_work+0x284/0x728)
[<c003ad2c>] (process_one_work) from [<c003b1fc>] (worker_thread+0x2c/0x52c)
[<c003b1fc>] (worker_thread) from [<c0042df8>] (kthread+0x154/0x180)
[<c0042df8>] (kthread) from [<c0008530>] (ret_from_fork+0x14/0x24)
Exception stack(0xc78ebfb0 to 0xc78ebff8)
bfa0:                                     00000000 00000000 00000000 00000000
bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
------------[ cut here ]------------

---
# bad: [ca0e494af5edb59002665bf12871e94b4163a257] Add linux-next specific files for 20200715
# good: [11ba468877bb23f28956a35e896356252d63c983] Linux 5.8-rc5
git bisect start 'HEAD' 'v5.8-rc5'
# good: [8a21074e4724ce86b4e1a8fdc271081a86303014] Merge remote-tracking branch 'crypto/master'
git bisect good 8a21074e4724ce86b4e1a8fdc271081a86303014
# good: [71f459b5cd96681f615e5c920b3a38ab89e8b5f1] Merge remote-tracking branch 'audit/next'
git bisect good 71f459b5cd96681f615e5c920b3a38ab89e8b5f1
# bad: [5a310f73f874211ee7aa570b308d50def85aff1c] Merge remote-tracking branch 'thunderbolt/next' git bisect bad 5a310f73f874211ee7aa570b308d50def85aff1c
# good: [db01f1b1e840f46bdd64c6c846e7b3246bdff55a] Merge remote-tracking branch 'kvm/linux-next' git bisect good db01f1b1e840f46bdd64c6c846e7b3246bdff55a
# bad: [89947fbc4b34bfdef7ade2b26274e78b32d4e39f] Merge remote-tracking branch 'usb/usb-next'
git bisect bad 89947fbc4b34bfdef7ade2b26274e78b32d4e39f
# bad: [c845fc08689b52c05c0201614ac89f7978caa169] Merge remote-tracking branch 'driver-core/driver-core-next' git bisect bad c845fc08689b52c05c0201614ac89f7978caa169
# good: [4621ddca720ceb9b45725acd428e0389e3c0b59b] Merge remote-tracking branch 'drivers-x86/for-next' git bisect good 4621ddca720ceb9b45725acd428e0389e3c0b59b
# good: [fac3dee864e971dff81dd4969c361b2b373109c0] Merge remote-tracking branch 'leds/for-next'
git bisect good fac3dee864e971dff81dd4969c361b2b373109c0
# good: [55c7549819e438f40a3ef1d8ac5c38b73390bcb7] scripts: add dummy report mode to add_namespace.cocci git bisect good 55c7549819e438f40a3ef1d8ac5c38b73390bcb7
# good: [fe940d7362e6d7a5e5086581d0462e00ba766992] driver core: Drop mention of obsolete bus rwsem from kernel-doc git bisect good fe940d7362e6d7a5e5086581d0462e00ba766992
# bad: [da6d647598a6d182eb6a0344a7b14ae005244399] driver core: Add waiting_for_supplier sysfs file for devices git bisect bad da6d647598a6d182eb6a0344a7b14ae005244399
# bad: [8fd456ec0cf03875908d6b67c1cd20cf0a7b4474] driver core: Add state_synced sysfs file for devices that support it git bisect bad 8fd456ec0cf03875908d6b67c1cd20cf0a7b4474
# bad: [287905e68dd29873bcb7986a8290cd1e4cfde600] driver core: Expose device link details in sysfs git bisect bad 287905e68dd29873bcb7986a8290cd1e4cfde600

# first bad commit: [287905e68dd29873bcb7986a8290cd1e4cfde600] driver core: Expose device link details in sysfs
#3 lore raw

Re: [PATCH v3 1/3] driver core: Expose device link details in sysfs

Quoted 11 lines · Guenter Roeck

Hi,

Quoted 7 lines · Saravana Kannan

It's helpful to be able to look at device link details from sysfs. So, expose it in sysfs.

Say device-A is supplier of device-B. These are the additional files this patch would create:

Quoted 5 lines
Quoted 1 line

This patch creates a lot of noise in linux-next when booting various qemu images. Example log output as well as bisect results below. Reverting the patch fixes the problem.

Thanks for reporting it. Marek reported it last night too. I'll take a look at this before the end of the week (probably tomorrow).

-Saravana

#4 lore raw

PATCH v3 2/3 driver core: Add state_synced sysfs file for devices that support it

In reply to Saravana Kannan

This can be used to check if a device supports sync_state() callbacks and therefore keeps resources left on by the bootloader enabled till all its consumers have probed.

This can also be used to check if sync_state() has been called for a device or whether it is still trying to keep resources enabled because they were left enabled by the bootloader and all its consumers haven't probed yet.

---
 .../ABI/testing/sysfs-devices-state_synced    | 24 +++++++++++++++++++
 drivers/base/dd.c                             | 22 +++++++++++++++++
 2 files changed, 46 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-state_synced
Documentation/ABI/testing/sysfs-devices-state_synced new file +24 -0
new file mode 100644
index 000000000000..0c922d7d02fc
@@ -0,0 +1,24 @@
+What:		/sys/devices/.../state_synced
+Date:		May 2020
+Contact:	Saravana Kannan <[email protected]>
+Description:
+		The /sys/devices/.../state_synced attribute is only present for
+		devices whose bus types or driver provides the .sync_state()
+		callback. The number read from it (0 or 1) reflects the value
+		of the device's 'state_synced' field. A value of 0 means the
+		.sync_state() callback hasn't been called yet. A value of 1
+		means the .sync_state() callback has been called.
+
+		Generally, if a device has sync_state() support and has some of
+		the resources it provides enabled at the time the kernel starts
+		(Eg: enabled by hardware reset or bootloader or anything that
+		run before the kernel starts), then it'll keep those resources
+		enabled and in a state that's compatible with the state they
+		were in at the start of the kernel. The device will stop doing
+		this only when the sync_state() callback has been called --
+		which happens only when all its consumer devices are registered
+		and have probed successfully. Resources that were left disabled
+		at the time the kernel starts are not affected or limited in
+		any way by sync_state() callbacks.
+
+
drivers/base/dd.c +22 -0
index 9a1d940342ac..7c04b88daac3 100644
@@ -463,6 +463,18 @@ static void driver_deferred_probe_add_trigger(struct device *dev,
 		driver_deferred_probe_trigger();
 }
 
+static ssize_t state_synced_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	bool val;
+
+	device_lock(dev);
+	val = dev->state_synced;
+	device_unlock(dev);
+	return sprintf(buf, "%u\n", val);
+}
+static DEVICE_ATTR_RO(state_synced);
+
 static int really_probe(struct device *dev, struct device_driver *drv)
 {
 	int ret = -EPROBE_DEFER;
@@ -536,9 +548,16 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 		goto dev_groups_failed;
 	}
 
+	if (dev_has_sync_state(dev) &&
+	    device_create_file(dev, &dev_attr_state_synced)) {
+		dev_err(dev, "state_synced sysfs add failed\n");
+		goto dev_sysfs_state_synced_failed;
+	}
+
 	if (test_remove) {
 		test_remove = false;
 
+		device_remove_file(dev, &dev_attr_state_synced);
 		device_remove_groups(dev, drv->dev_groups);
 
 		if (dev->bus->remove)
@@ -568,6 +587,8 @@ static int really_probe(struct device *dev, struct device_driver *drv)
 		 drv->bus->name, __func__, dev_name(dev), drv->name);
 	goto done;
 
+dev_sysfs_state_synced_failed:
+	device_remove_groups(dev, drv->dev_groups);
 dev_groups_failed:
 	if (dev->bus->remove)
 		dev->bus->remove(dev);
@@ -1105,6 +1126,7 @@ static void __device_release_driver(struct device *dev, struct device *parent)
 
 		pm_runtime_put_sync(dev);
 
+		device_remove_file(dev, &dev_attr_state_synced);
 		device_remove_groups(dev, drv->dev_groups);
 
 		if (dev->bus && dev->bus->remove)
Signature · 1 line
2.27.0.rc0.183.gde8f92d652-goog
#5 lore raw

PATCH v3 3/3 driver core: Add waiting_for_supplier sysfs file for devices

In reply to Saravana Kannan

This would be useful to check if a device is not probing because it's waiting for a supplier to be added and then linked to before it can probe.

To reduce sysfs clutter, this file is added only if it can ever be 1. So, if fw_devlink is disabled or set to permissive, this file is not added. Also, this file is removed once the device probes as it's no longer relevant.

---
 .../sysfs-devices-waiting_for_supplier        | 17 ++++++++++++
 drivers/base/core.c                           | 26 +++++++++++++++++++
 2 files changed, 43 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-waiting_for_supplier
Documentation/ABI/testing/sysfs-devices-waiting_for_supplier new file +17 -0
new file mode 100644
index 000000000000..59d073d20db6
@@ -0,0 +1,17 @@
+What:		/sys/devices/.../waiting_for_supplier
+Date:		May 2020
+Contact:	Saravana Kannan <[email protected]>
+Description:
+		The /sys/devices/.../waiting_for_supplier attribute is only
+		present when fw_devlink kernel command line option is enabled
+		and is set to something stricter than "permissive".  It is
+		removed once a device probes successfully (because the
+		information is no longer relevant). The number read from it (0
+		or 1) reflects whether the device is waiting for one or more
+		suppliers to be added and then linked to using device links
+		before the device can probe.
+
+		A value of 0 means the device is not waiting for any suppliers
+		to be added before it can probe.  A value of 1 means the device
+		is waiting for one or more suppliers to be added before it can
+		probe.
drivers/base/core.c +26 -0
index 81c8ef088d3a..dfd4e94ef790 100644
@@ -1033,6 +1033,22 @@ static void device_link_drop_managed(struct device_link *link)
 	kref_put(&link->kref, __device_link_del);
 }
 
+static ssize_t waiting_for_supplier_show(struct device *dev,
+					 struct device_attribute *attr,
+					 char *buf)
+{
+	bool val;
+
+	device_lock(dev);
+	mutex_lock(&wfs_lock);
+	val = !list_empty(&dev->links.needs_suppliers)
+	      && dev->links.need_for_probe;
+	mutex_unlock(&wfs_lock);
+	device_unlock(dev);
+	return sprintf(buf, "%u\n", val);
+}
+static DEVICE_ATTR_RO(waiting_for_supplier);
+
 /**
  * device_links_driver_bound - Update device links after probing its driver.
  * @dev: Device to update the links for.
@@ -1057,6 +1073,7 @@ void device_links_driver_bound(struct device *dev)
 	mutex_lock(&wfs_lock);
 	list_del_init(&dev->links.needs_suppliers);
 	mutex_unlock(&wfs_lock);
+	device_remove_file(dev, &dev_attr_waiting_for_supplier);
 
 	device_links_write_lock();
 
@@ -2126,8 +2143,16 @@ static int device_add_attrs(struct device *dev)
 			goto err_remove_dev_groups;
 	}
 
+	if (fw_devlink_flags && !fw_devlink_is_permissive()) {
+		error = device_create_file(dev, &dev_attr_waiting_for_supplier);
+		if (error)
+			goto err_remove_dev_online;
+	}
+
 	return 0;
 
+ err_remove_dev_online:
+	device_remove_file(dev, &dev_attr_online);
  err_remove_dev_groups:
 	device_remove_groups(dev, dev->groups);
  err_remove_type_groups:
@@ -2145,6 +2170,7 @@ static void device_remove_attrs(struct device *dev)
 	struct class *class = dev->class;
 	const struct device_type *type = dev->type;
 
+	device_remove_file(dev, &dev_attr_waiting_for_supplier);
 	device_remove_file(dev, &dev_attr_online);
 	device_remove_groups(dev, dev->groups);
 
Signature · 1 line
2.27.0.rc0.183.gde8f92d652-goog
#6 lore raw
In reply to Saravana Kannan
Quoted 40 lines · Saravana Kannan

With fw_devlink and with sync_state() callback features, there's a lot of device/device link related information that's not available in sysfs.

Exposing these details to user space can be very useful in understanding suspend/resume issues, runtime pm issues, probing issues, figuring out the modules that'd be needed for first stage init, etc. In fact, an earlier verion of this series was very helpful in debugging and validating the recent memory leak fix[1].

This series is based on driver-core-next and [1] cherry-picked on top of it.

[1] - https://lore.kernel.org/lkml/[email protected]/

v1->v2:
Patch 1/4
- New patch
Patch 2/4
- Fixed the warnings I saw before that were related to incorrect sysfs removal code when a device link is deleted.
- Fixed error handling in device_link_add()
- Split up flags into more meaningful files.
- Added status file.
Patch 3/4
- Fixed error handling that Greg pointed out before. Patch 4/4
- New patch

v2->v3:
- Dropped patch 1/4 from v2 since it was already picked up. So numbering has shifted.
Patch 1/3
- Added documentation

Saravana Kannan (3):
driver core: Expose device link details in sysfs
driver core: Add state_synced sysfs file for devices that support it driver core: Add waiting_for_supplier sysfs file for devices

Quoted 14 lines
 Documentation/ABI/testing/sysfs-class-devlink | 126 ++++++++++
 .../ABI/testing/sysfs-devices-consumer        |   8 +
 .../ABI/testing/sysfs-devices-state_synced    |  24 ++
 .../ABI/testing/sysfs-devices-supplier        |   8 +
 .../sysfs-devices-waiting_for_supplier        |  17 ++
 drivers/base/core.c                           | 237 +++++++++++++++++-
 drivers/base/dd.c                             |  22 ++
 include/linux/device.h                        |  58 ++---
 8 files changed, 464 insertions(+), 36 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-class-devlink
 create mode 100644 Documentation/ABI/testing/sysfs-devices-consumer
 create mode 100644 Documentation/ABI/testing/sysfs-devices-state_synced
 create mode 100644 Documentation/ABI/testing/sysfs-devices-supplier
 create mode 100644 Documentation/ABI/testing/sysfs-devices-waiting_for_supplier

Friendly reminder.

-Saravana

#7 lore raw
Quoted 56 lines · Saravana Kannan
Quoted 40 lines · Saravana Kannan

With fw_devlink and with sync_state() callback features, there's a lot of device/device link related information that's not available in sysfs.

Exposing these details to user space can be very useful in understanding suspend/resume issues, runtime pm issues, probing issues, figuring out the modules that'd be needed for first stage init, etc. In fact, an earlier verion of this series was very helpful in debugging and validating the recent memory leak fix[1].

This series is based on driver-core-next and [1] cherry-picked on top of it.

[1] - https://lore.kernel.org/lkml/[email protected]/

v1->v2:
Patch 1/4
- New patch
Patch 2/4
- Fixed the warnings I saw before that were related to incorrect sysfs removal code when a device link is deleted.
- Fixed error handling in device_link_add()
- Split up flags into more meaningful files.
- Added status file.
Patch 3/4
- Fixed error handling that Greg pointed out before. Patch 4/4
- New patch

v2->v3:
- Dropped patch 1/4 from v2 since it was already picked up. So numbering has shifted.
Patch 1/3
- Added documentation

Saravana Kannan (3):
driver core: Expose device link details in sysfs
driver core: Add state_synced sysfs file for devices that support it driver core: Add waiting_for_supplier sysfs file for devices

Quoted 14 lines
 Documentation/ABI/testing/sysfs-class-devlink | 126 ++++++++++
 .../ABI/testing/sysfs-devices-consumer        |   8 +
 .../ABI/testing/sysfs-devices-state_synced    |  24 ++
 .../ABI/testing/sysfs-devices-supplier        |   8 +
 .../sysfs-devices-waiting_for_supplier        |  17 ++
 drivers/base/core.c                           | 237 +++++++++++++++++-
 drivers/base/dd.c                             |  22 ++
 include/linux/device.h                        |  58 ++---
 8 files changed, 464 insertions(+), 36 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-class-devlink
 create mode 100644 Documentation/ABI/testing/sysfs-devices-consumer
 create mode 100644 Documentation/ABI/testing/sysfs-devices-state_synced
 create mode 100644 Documentation/ABI/testing/sysfs-devices-supplier
 create mode 100644 Documentation/ABI/testing/sysfs-devices-waiting_for_supplier
Quoted 1 line

Friendly reminder.

Looks semi-sane, but it's too close to the merge window at the moment for me to take this. If there's no objections by the time 5.8-rc1 is out, I'll queue it up in my tree for 5.9-rc1.

thanks,

greg k-h

#8 lore raw
Quoted 3 lines · Greg Kroah-Hartman

Looks semi-sane, but it's too close to the merge window at the moment for me to take this. If there's no objections by the time 5.8-rc1 is out, I'll queue it up in my tree for 5.9-rc1.

Another friendly reminder :)

-Saravana

#9 lore raw
Quoted 8 lines · Saravana Kannan
Quoted 3 lines · Greg Kroah-Hartman

Looks semi-sane, but it's too close to the merge window at the moment for me to take this. If there's no objections by the time 5.8-rc1 is out, I'll queue it up in my tree for 5.9-rc1.

Another friendly reminder :)

*nudge* *nudge*

-Saravana

#10 lore raw
Quoted 11 lines · Saravana Kannan
Quoted 8 lines · Saravana Kannan
Quoted 3 lines · Greg Kroah-Hartman

Looks semi-sane, but it's too close to the merge window at the moment for me to take this. If there's no objections by the time 5.8-rc1 is out, I'll queue it up in my tree for 5.9-rc1.

Another friendly reminder :)

*nudge* *nudge*

Looks sane, given no objections, let's see what linux-next thinks about it...

#11 lore raw
Quoted 10 lines · Greg Kroah-Hartman
Quoted 8 lines · Saravana Kannan
Quoted 7 lines · Saravana Kannan
Quoted 3 lines · Greg Kroah-Hartman

Looks semi-sane, but it's too close to the merge window at the moment for me to take this. If there's no objections by the time 5.8-rc1 is out, I'll queue it up in my tree for 5.9-rc1.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Another friendly reminder :)

*nudge* *nudge*

Looks sane, given no objections, let's see what linux-next thinks about it...

Thanks!

-Saravana

#12 lore raw
In reply to Greg Kroah-Hartman

Hi Greg and Saravana,

Quoted 11 lines · Greg Kroah-Hartman
Quoted 8 lines · Saravana Kannan
Quoted 6 lines · Saravana Kannan
Quoted 3 lines · Greg Kroah-Hartman

Looks semi-sane, but it's too close to the merge window at the moment for me to take this. If there's no objections by the time 5.8-rc1 is out, I'll queue it up in my tree for 5.9-rc1.

Another friendly reminder :)

*nudge* *nudge*

Looks sane, given no objections, let's see what linux-next thinks about it...

linux-next is not very happy from this patchset... Starting from next-20200713 I see a few new issues on various Samsung Exynos based boards. Here are examples from Exynos4412-based Odroid U3 board (ARM 32bit, kernel compiled from exynos_defconfig):

BUG: sleeping function called from invalid context at 

kernel/locking/mutex.c:935
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 12, name: kworker/0:1 2 locks held by kworker/0:1/12:
 #0: ee8074a8 ((wq_completion)rcu_gp){+.+.}-{0:0}, at:

process_one_work+0x174/0x7dc
  #1: ee921f20 ((work_completion)(&sdp->work)){+.+.}-{0:0}, at: 
process_one_work+0x174/0x7dc
Preemption disabled at:
[<c01b10f0>] srcu_invoke_callbacks+0xc0/0x154
CPU: 0 PID: 12 Comm: kworker/0:1 Tainted: G        W 
5.8.0-rc3-00022-g287905e68dd2 #8753
Hardware name: Samsung Exynos (Flattened Device Tree)
Workqueue: rcu_gp srcu_invoke_callbacks
[<c011184c>] (unwind_backtrace) from [<c010d250>] (show_stack+0x10/0x14)
[<c010d250>] (show_stack) from [<c0517f04>] (dump_stack+0xbc/0xe8)
[<c0517f04>] (dump_stack) from [<c0159188>] (___might_sleep+0x288/0x2d8)
[<c0159188>] (___might_sleep) from [<c0abbd1c>] (__mutex_lock+0x48/0xb18)
[<c0abbd1c>] (__mutex_lock) from [<c0abc808>] (mutex_lock_nested+0x1c/0x24)
[<c0abc808>] (mutex_lock_nested) from [<c064590c>] (device_del+0x30/0x39c)
[<c064590c>] (device_del) from [<c0645c9c>] (device_unregister+0x24/0x64)

[<c0645c9c>] (device_unregister) from [<c01b10fc>] (srcu_invoke_callbacks+0xcc/0x154)
[<c01b10fc>] (srcu_invoke_callbacks) from [<c01493c4>] (process_one_work+0x234/0x7dc)
[<c01493c4>] (process_one_work) from [<c01499b0>] (worker_thread+0x44/0x51c) [<c01499b0>] (worker_thread) from [<c0150bf4>] (kthread+0x158/0x1a0) [<c0150bf4>] (kthread) from [<c0100114>] (ret_from_fork+0x14/0x20)

Exception stack(0xee921fb0 to 0xee921ff8)


BUG: scheduling while atomic: kworker/0:0/5/0x00000201
3 locks held by kworker/0:0/5:
  #0: ee8074a8 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: 
process_one_work+0x174/0x7dc
  #1: ee907f20 ((work_completion)(&sdp->work)){+.+.}-{0:0}, at: 
process_one_work+0x174/0x7dc
  #2: c11367e4 (kernfs_mutex){+.+.}-{3:3}, at: 
kernfs_remove_by_name_ns+0x24/0x94
Modules linked in:
Preemption disabled at:
[<c01b10f0>] srcu_invoke_callbacks+0xc0/0x154
CPU: 0 PID: 5 Comm: kworker/0:0 Tainted: G        W 
5.8.0-rc3-00022-g287905e68dd2 #8753
Hardware name: Samsung Exynos (Flattened Device Tree)
Workqueue: rcu_gp srcu_invoke_callbacks
[<c011184c>] (unwind_backtrace) from [<c010d250>] (show_stack+0x10/0x14)
[<c010d250>] (show_stack) from [<c0517f04>] (dump_stack+0xbc/0xe8)
[<c0517f04>] (dump_stack) from [<c0158da0>] (__schedule_bug+0x88/0xe0)
[<c0158da0>] (__schedule_bug) from [<c0ab7f88>] (__schedule+0x5d0/0x83c)
[<c0ab7f88>] (__schedule) from [<c0ab8258>] (schedule+0x64/0x124)

[<c0ab8258>] (schedule) from [<c0ab8768>] (schedule_preempt_disabled+0x14/0x20)
[<c0ab8768>] (schedule_preempt_disabled) from [<c0abc1ec>] (__mutex_lock+0x518/0xb18)
[<c0abc1ec>] (__mutex_lock) from [<c0abc808>] (mutex_lock_nested+0x1c/0x24) [<c0abc808>] (mutex_lock_nested) from [<c0362fc0>] (kernfs_remove_by_name_ns+0x24/0x94)
[<c0362fc0>] (kernfs_remove_by_name_ns) from [<c0644eac>] (device_remove_class_symlinks+0x70/0x94)
[<c0644eac>] (device_remove_class_symlinks) from [<c0645994>] (device_del+0xb8/0x39c)
[<c0645994>] (device_del) from [<c0645c9c>] (device_unregister+0x24/0x64) [<c0645c9c>] (device_unregister) from [<c01b10fc>] (srcu_invoke_callbacks+0xcc/0x154)
[<c01b10fc>] (srcu_invoke_callbacks) from [<c01493c4>] (process_one_work+0x234/0x7dc)
[<c01493c4>] (process_one_work) from [<c01499b0>] (worker_thread+0x44/0x51c) [<c01499b0>] (worker_thread) from [<c0150bf4>] (kthread+0x158/0x1a0) [<c0150bf4>] (kthread) from [<c0100114>] (ret_from_fork+0x14/0x20)

Exception stack(0xee907fb0 to 0xee907ff8)

sysfs: cannot create duplicate filename
'/devices/virtual/devlink/regulator.28:cpu0'

------------[ cut here ]------------
WARNING: CPU: 0 PID: 5 at kernel/sched/core.c:3897 
preempt_count_sub+0x104/0x108
DEBUG_LOCKS_WARN_ON(val > preempt_count())
Modules linked in:
CPU: 0 PID: 5 Comm: kworker/0:0 Tainted: G        W 
5.8.0-rc3-00022-g287905e68dd2 #8753
Hardware name: Samsung Exynos (Flattened Device Tree)
Workqueue: rcu_gp srcu_invoke_callbacks
[<c011184c>] (unwind_backtrace) from [<c010d250>] (show_stack+0x10/0x14)
[<c010d250>] (show_stack) from [<c0517f04>] (dump_stack+0xbc/0xe8)
[<c0517f04>] (dump_stack) from [<c01270a8>] (__warn+0xf0/0x108)
[<c01270a8>] (__warn) from [<c0127134>] (warn_slowpath_fmt+0x74/0xb8)

[<c0127134>] (warn_slowpath_fmt) from [<c0158efc>] (preempt_count_sub+0x104/0x108)
[<c0158efc>] (preempt_count_sub) from [<c012fee4>] (__local_bh_enable_ip+0x74/0x16c)
[<c012fee4>] (__local_bh_enable_ip) from [<c01b1108>] (srcu_invoke_callbacks+0xd8/0x154)
[<c01b1108>] (srcu_invoke_callbacks) from [<c01493c4>] (process_one_work+0x234/0x7dc)
[<c01493c4>] (process_one_work) from [<c01499b0>] (worker_thread+0x44/0x51c) [<c01499b0>] (worker_thread) from [<c0150bf4>] (kthread+0x158/0x1a0) [<c0150bf4>] (kthread) from [<c0100114>] (ret_from_fork+0x14/0x20)

Exception stack(0xee907fb0 to 0xee907ff8)

7fa0:                                     00000000 00000000 00000000 00000000
7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
7fe0: 00000000 00000000 00000000 00000000 00000013 00000000

irq event stamp: 2274
hardirqs last  enabled at (2273): [<c02b3274>] __slab_free+0x2b0/0x480
hardirqs last disabled at (2274): [<c012fecc>] 
__local_bh_enable_ip+0x5c/0x16c
softirqs last  enabled at (2184): [<c010174c>] __do_softirq+0x50c/0x608
softirqs last disabled at (2056): [<c01b10e4>] 
srcu_invoke_callbacks+0xb4/0x154
---[ end trace eaa99e7b5e453175 ]---

PU: 0 PID: 1 Comm: swapper/0 Tainted: G        W
5.8.0-rc3-00022-g287905e68dd2 #8753

Hardware name: Samsung Exynos (Flattened Device Tree)
[<c011184c>] (unwind_backtrace) from [<c010d250>] (show_stack+0x10/0x14)
[<c010d250>] (show_stack) from [<c0517f04>] (dump_stack+0xbc/0xe8)
[<c0517f04>] (dump_stack) from [<c0365474>] (sysfs_warn_dup+0x58/0x64)

[<c0365474>] (sysfs_warn_dup) from [<c03655a8>] (sysfs_create_dir_ns+0xe0/0xf4)
[<c03655a8>] (sysfs_create_dir_ns) from [<c051f218>] (kobject_add_internal+0xb8/0x370)
[<c051f218>] (kobject_add_internal) from [<c051f69c>] (kobject_add+0x5c/0xbc)
[<c051f69c>] (kobject_add) from [<c0647338>] (device_add+0x100/0x744) [<c0647338>] (device_add) from [<c0647b88>] (device_link_add+0x1f4/0x5d0) [<c0647b88>] (device_link_add) from [<c0582f60>] (_regulator_get+0xf4/0x280) [<c0582f60>] (_regulator_get) from [<c0803da4>] (dev_pm_opp_set_regulators+0xf8/0x210)
[<c0803da4>] (dev_pm_opp_set_regulators) from [<c080df80>] (cpufreq_init+0xb8/0x304)
[<c080df80>] (cpufreq_init) from [<c080aae0>] (cpufreq_online+0x2c8/0x9f8) [<c080aae0>] (cpufreq_online) from [<c080b2ac>] (cpufreq_add_dev+0x8c/0xc0) [<c080b2ac>] (cpufreq_add_dev) from [<c0649854>] (subsys_interface_register+0xa8/0xf4)
[<c0649854>] (subsys_interface_register) from [<c0808184>] (cpufreq_register_driver+0x178/0x294)
[<c0808184>] (cpufreq_register_driver) from [<c080ddac>] (dt_cpufreq_probe+0xd8/0x1f4)
[<c080ddac>] (dt_cpufreq_probe) from [<c064e048>] (platform_drv_probe+0x6c/0xa4)
[<c064e048>] (platform_drv_probe) from [<c064b664>] (really_probe+0x200/0x48c)
[<c064b664>] (really_probe) from [<c064ba58>] (driver_probe_device+0x78/0x1fc)
[<c064ba58>] (driver_probe_device) from [<c064be40>] (device_driver_attach+0x58/0x60)
[<c064be40>] (device_driver_attach) from [<c064bf24>] (__driver_attach+0xdc/0x174)
[<c064bf24>] (__driver_attach) from [<c0649470>] (bus_for_each_dev+0x68/0xb4)
[<c0649470>] (bus_for_each_dev) from [<c064a7a4>] (bus_add_driver+0x158/0x214)
[<c064a7a4>] (bus_add_driver) from [<c064cdf8>] (driver_register+0x78/0x110) [<c064cdf8>] (driver_register) from [<c0102378>] (do_one_initcall+0x8c/0x424)
[<c0102378>] (do_one_initcall) from [<c1001158>] (kernel_init_freeable+0x190/0x204)
[<c1001158>] (kernel_init_freeable) from [<c0ab7104>] (kernel_init+0x8/0x118)
[<c0ab7104>] (kernel_init) from [<c0100114>] (ret_from_fork+0x14/0x20)

Exception stack(0xee8f1fb0 to 0xee8f1ff8)
...
BUG: sleeping function called from invalid context at 

kernel/locking/mutex.c:935
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 5, name: kworker/0:0
INFO: lockdep is turned off.
Preemption disabled at:

[<c01b10f0>] srcu_invoke_callbacks+0xc0/0x154
CPU: 0 PID: 5 Comm: kworker/0:0 Tainted: G        W 
5.8.0-rc3-00022-g287905e68dd2 #8753
Hardware name: Samsung Exynos (Flattened Device Tree)
Workqueue: rcu_gp srcu_invoke_callbacks
[<c011184c>] (unwind_backtrace) from [<c010d250>] (show_stack+0x10/0x14)
[<c010d250>] (show_stack) from [<c0517f04>] (dump_stack+0xbc/0xe8)
[<c0517f04>] (dump_stack) from [<c0159188>] (___might_sleep+0x288/0x2d8)
[<c0159188>] (___might_sleep) from [<c0abbd1c>] (__mutex_lock+0x48/0xb18)
[<c0abbd1c>] (__mutex_lock) from [<c0abc808>] (mutex_lock_nested+0x1c/0x24)
[<c0abc808>] (mutex_lock_nested) from [<c064590c>] (device_del+0x30/0x39c)
[<c064590c>] (device_del) from [<c0645c9c>] (device_unregister+0x24/0x64)

[<c0645c9c>] (device_unregister) from [<c01b10fc>] (srcu_invoke_callbacks+0xcc/0x154)
[<c01b10fc>] (srcu_invoke_callbacks) from [<c01493c4>] (process_one_work+0x234/0x7dc)
[<c01493c4>] (process_one_work) from [<c01499b0>] (worker_thread+0x44/0x51c) [<c01499b0>] (worker_thread) from [<c0150bf4>] (kthread+0x158/0x1a0) [<c0150bf4>] (kthread) from [<c0100114>] (ret_from_fork+0x14/0x20)

Exception stack(0xee907fb0 to 0xee907ff8)

...
kobject_add_internal failed for regulator.28:cpu0 with -EEXIST, don't try to register things with the same name in the same directory.

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#13 lore raw
Quoted 12 lines · Marek Szyprowski

Hi Greg and Saravana,

Quoted 8 lines · Greg Kroah-Hartman
Quoted 7 lines · Saravana Kannan
Quoted 6 lines · Saravana Kannan
Quoted 3 lines · Greg Kroah-Hartman

Looks semi-sane, but it's too close to the merge window at the moment for me to take this. If there's no objections by the time 5.8-rc1 is out, I'll queue it up in my tree for 5.9-rc1.

Another friendly reminder :)

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

*nudge* *nudge*

Looks sane, given no objections, let's see what linux-next thinks about it...

linux-next is not very happy from this patchset... Starting from next-20200713 I see a few new issues on various Samsung Exynos based boards. Here are examples from Exynos4412-based Odroid U3 board (ARM 32bit, kernel compiled from exynos_defconfig):

Thanks for the bug reports.

Quoted 10 lines
BUG: sleeping function called from invalid context at

kernel/locking/mutex.c:935
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 12, name: kworker/0:1 2 locks held by kworker/0:1/12:
#0: ee8074a8 ((wq_completion)rcu_gp){+.+.}-{0:0}, at:

process_one_work+0x174/0x7dc
  #1: ee921f20 ((work_completion)(&sdp->work)){+.+.}-{0:0}, at:
process_one_work+0x174/0x7dc

Preemption disabled at:
[<c01b10f0>] srcu_invoke_callbacks+0xc0/0x154

Sigh... probably some SRCU screw up when the device link is deleted. I'll look at it by the end of this week. If you don't mind, what SRCU debug config caught this for you? That way, I can reproduce it on my end.

-Saravana

Quoted 188 lines
CPU: 0 PID: 12 Comm: kworker/0:1 Tainted: G        W
5.8.0-rc3-00022-g287905e68dd2 #8753
Hardware name: Samsung Exynos (Flattened Device Tree)
Workqueue: rcu_gp srcu_invoke_callbacks
[<c011184c>] (unwind_backtrace) from [<c010d250>] (show_stack+0x10/0x14)
[<c010d250>] (show_stack) from [<c0517f04>] (dump_stack+0xbc/0xe8)
[<c0517f04>] (dump_stack) from [<c0159188>] (___might_sleep+0x288/0x2d8)
[<c0159188>] (___might_sleep) from [<c0abbd1c>] (__mutex_lock+0x48/0xb18)
[<c0abbd1c>] (__mutex_lock) from [<c0abc808>] (mutex_lock_nested+0x1c/0x24)
[<c0abc808>] (mutex_lock_nested) from [<c064590c>] (device_del+0x30/0x39c)
[<c064590c>] (device_del) from [<c0645c9c>] (device_unregister+0x24/0x64)

[<c0645c9c>] (device_unregister) from [<c01b10fc>] (srcu_invoke_callbacks+0xcc/0x154)
[<c01b10fc>] (srcu_invoke_callbacks) from [<c01493c4>] (process_one_work+0x234/0x7dc)
[<c01493c4>] (process_one_work) from [<c01499b0>] (worker_thread+0x44/0x51c) [<c01499b0>] (worker_thread) from [<c0150bf4>] (kthread+0x158/0x1a0) [<c0150bf4>] (kthread) from [<c0100114>] (ret_from_fork+0x14/0x20)

Exception stack(0xee921fb0 to 0xee921ff8)


BUG: scheduling while atomic: kworker/0:0/5/0x00000201
3 locks held by kworker/0:0/5:
  #0: ee8074a8 ((wq_completion)rcu_gp){+.+.}-{0:0}, at:
process_one_work+0x174/0x7dc
  #1: ee907f20 ((work_completion)(&sdp->work)){+.+.}-{0:0}, at:
process_one_work+0x174/0x7dc
  #2: c11367e4 (kernfs_mutex){+.+.}-{3:3}, at:
kernfs_remove_by_name_ns+0x24/0x94
Modules linked in:
Preemption disabled at:
[<c01b10f0>] srcu_invoke_callbacks+0xc0/0x154
CPU: 0 PID: 5 Comm: kworker/0:0 Tainted: G        W
5.8.0-rc3-00022-g287905e68dd2 #8753
Hardware name: Samsung Exynos (Flattened Device Tree)
Workqueue: rcu_gp srcu_invoke_callbacks
[<c011184c>] (unwind_backtrace) from [<c010d250>] (show_stack+0x10/0x14)
[<c010d250>] (show_stack) from [<c0517f04>] (dump_stack+0xbc/0xe8)
[<c0517f04>] (dump_stack) from [<c0158da0>] (__schedule_bug+0x88/0xe0)
[<c0158da0>] (__schedule_bug) from [<c0ab7f88>] (__schedule+0x5d0/0x83c)
[<c0ab7f88>] (__schedule) from [<c0ab8258>] (schedule+0x64/0x124)

[<c0ab8258>] (schedule) from [<c0ab8768>] (schedule_preempt_disabled+0x14/0x20)
[<c0ab8768>] (schedule_preempt_disabled) from [<c0abc1ec>] (__mutex_lock+0x518/0xb18)
[<c0abc1ec>] (__mutex_lock) from [<c0abc808>] (mutex_lock_nested+0x1c/0x24) [<c0abc808>] (mutex_lock_nested) from [<c0362fc0>] (kernfs_remove_by_name_ns+0x24/0x94)
[<c0362fc0>] (kernfs_remove_by_name_ns) from [<c0644eac>] (device_remove_class_symlinks+0x70/0x94)
[<c0644eac>] (device_remove_class_symlinks) from [<c0645994>] (device_del+0xb8/0x39c)
[<c0645994>] (device_del) from [<c0645c9c>] (device_unregister+0x24/0x64) [<c0645c9c>] (device_unregister) from [<c01b10fc>] (srcu_invoke_callbacks+0xcc/0x154)
[<c01b10fc>] (srcu_invoke_callbacks) from [<c01493c4>] (process_one_work+0x234/0x7dc)
[<c01493c4>] (process_one_work) from [<c01499b0>] (worker_thread+0x44/0x51c) [<c01499b0>] (worker_thread) from [<c0150bf4>] (kthread+0x158/0x1a0) [<c0150bf4>] (kthread) from [<c0100114>] (ret_from_fork+0x14/0x20)

Exception stack(0xee907fb0 to 0xee907ff8)

sysfs: cannot create duplicate filename
'/devices/virtual/devlink/regulator.28:cpu0'

------------[ cut here ]------------
WARNING: CPU: 0 PID: 5 at kernel/sched/core.c:3897
preempt_count_sub+0x104/0x108
DEBUG_LOCKS_WARN_ON(val > preempt_count())
Modules linked in:
CPU: 0 PID: 5 Comm: kworker/0:0 Tainted: G        W
5.8.0-rc3-00022-g287905e68dd2 #8753
Hardware name: Samsung Exynos (Flattened Device Tree)
Workqueue: rcu_gp srcu_invoke_callbacks
[<c011184c>] (unwind_backtrace) from [<c010d250>] (show_stack+0x10/0x14)
[<c010d250>] (show_stack) from [<c0517f04>] (dump_stack+0xbc/0xe8)
[<c0517f04>] (dump_stack) from [<c01270a8>] (__warn+0xf0/0x108)
[<c01270a8>] (__warn) from [<c0127134>] (warn_slowpath_fmt+0x74/0xb8)

[<c0127134>] (warn_slowpath_fmt) from [<c0158efc>] (preempt_count_sub+0x104/0x108)
[<c0158efc>] (preempt_count_sub) from [<c012fee4>] (__local_bh_enable_ip+0x74/0x16c)
[<c012fee4>] (__local_bh_enable_ip) from [<c01b1108>] (srcu_invoke_callbacks+0xd8/0x154)
[<c01b1108>] (srcu_invoke_callbacks) from [<c01493c4>] (process_one_work+0x234/0x7dc)
[<c01493c4>] (process_one_work) from [<c01499b0>] (worker_thread+0x44/0x51c) [<c01499b0>] (worker_thread) from [<c0150bf4>] (kthread+0x158/0x1a0) [<c0150bf4>] (kthread) from [<c0100114>] (ret_from_fork+0x14/0x20)

Exception stack(0xee907fb0 to 0xee907ff8)

7fa0: 00000000 00000000 00000000
00000000
7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
7fe0: 00000000 00000000 00000000 00000000 00000013 00000000

irq event stamp: 2274
hardirqs last  enabled at (2273): [<c02b3274>] __slab_free+0x2b0/0x480
hardirqs last disabled at (2274): [<c012fecc>]
__local_bh_enable_ip+0x5c/0x16c
softirqs last  enabled at (2184): [<c010174c>] __do_softirq+0x50c/0x608
softirqs last disabled at (2056): [<c01b10e4>]
srcu_invoke_callbacks+0xb4/0x154
---[ end trace eaa99e7b5e453175 ]---

PU: 0 PID: 1 Comm: swapper/0 Tainted: G W
5.8.0-rc3-00022-g287905e68dd2 #8753

Hardware name: Samsung Exynos (Flattened Device Tree)
[<c011184c>] (unwind_backtrace) from [<c010d250>] (show_stack+0x10/0x14)
[<c010d250>] (show_stack) from [<c0517f04>] (dump_stack+0xbc/0xe8)
[<c0517f04>] (dump_stack) from [<c0365474>] (sysfs_warn_dup+0x58/0x64)

[<c0365474>] (sysfs_warn_dup) from [<c03655a8>] (sysfs_create_dir_ns+0xe0/0xf4)
[<c03655a8>] (sysfs_create_dir_ns) from [<c051f218>] (kobject_add_internal+0xb8/0x370)
[<c051f218>] (kobject_add_internal) from [<c051f69c>] (kobject_add+0x5c/0xbc)
[<c051f69c>] (kobject_add) from [<c0647338>] (device_add+0x100/0x744) [<c0647338>] (device_add) from [<c0647b88>] (device_link_add+0x1f4/0x5d0) [<c0647b88>] (device_link_add) from [<c0582f60>] (_regulator_get+0xf4/0x280) [<c0582f60>] (_regulator_get) from [<c0803da4>] (dev_pm_opp_set_regulators+0xf8/0x210)
[<c0803da4>] (dev_pm_opp_set_regulators) from [<c080df80>] (cpufreq_init+0xb8/0x304)
[<c080df80>] (cpufreq_init) from [<c080aae0>] (cpufreq_online+0x2c8/0x9f8) [<c080aae0>] (cpufreq_online) from [<c080b2ac>] (cpufreq_add_dev+0x8c/0xc0) [<c080b2ac>] (cpufreq_add_dev) from [<c0649854>] (subsys_interface_register+0xa8/0xf4)
[<c0649854>] (subsys_interface_register) from [<c0808184>] (cpufreq_register_driver+0x178/0x294)
[<c0808184>] (cpufreq_register_driver) from [<c080ddac>] (dt_cpufreq_probe+0xd8/0x1f4)
[<c080ddac>] (dt_cpufreq_probe) from [<c064e048>] (platform_drv_probe+0x6c/0xa4)
[<c064e048>] (platform_drv_probe) from [<c064b664>] (really_probe+0x200/0x48c)
[<c064b664>] (really_probe) from [<c064ba58>] (driver_probe_device+0x78/0x1fc)
[<c064ba58>] (driver_probe_device) from [<c064be40>] (device_driver_attach+0x58/0x60)
[<c064be40>] (device_driver_attach) from [<c064bf24>] (__driver_attach+0xdc/0x174)
[<c064bf24>] (__driver_attach) from [<c0649470>] (bus_for_each_dev+0x68/0xb4)
[<c0649470>] (bus_for_each_dev) from [<c064a7a4>] (bus_add_driver+0x158/0x214)
[<c064a7a4>] (bus_add_driver) from [<c064cdf8>] (driver_register+0x78/0x110) [<c064cdf8>] (driver_register) from [<c0102378>] (do_one_initcall+0x8c/0x424)
[<c0102378>] (do_one_initcall) from [<c1001158>] (kernel_init_freeable+0x190/0x204)
[<c1001158>] (kernel_init_freeable) from [<c0ab7104>] (kernel_init+0x8/0x118)
[<c0ab7104>] (kernel_init) from [<c0100114>] (ret_from_fork+0x14/0x20)

Exception stack(0xee8f1fb0 to 0xee8f1ff8)
...
BUG: sleeping function called from invalid context at

kernel/locking/mutex.c:935
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 5, name: kworker/0:0
INFO: lockdep is turned off.
Preemption disabled at:

[<c01b10f0>] srcu_invoke_callbacks+0xc0/0x154
CPU: 0 PID: 5 Comm: kworker/0:0 Tainted: G        W
5.8.0-rc3-00022-g287905e68dd2 #8753
Hardware name: Samsung Exynos (Flattened Device Tree)
Workqueue: rcu_gp srcu_invoke_callbacks
[<c011184c>] (unwind_backtrace) from [<c010d250>] (show_stack+0x10/0x14)
[<c010d250>] (show_stack) from [<c0517f04>] (dump_stack+0xbc/0xe8)
[<c0517f04>] (dump_stack) from [<c0159188>] (___might_sleep+0x288/0x2d8)
[<c0159188>] (___might_sleep) from [<c0abbd1c>] (__mutex_lock+0x48/0xb18)
[<c0abbd1c>] (__mutex_lock) from [<c0abc808>] (mutex_lock_nested+0x1c/0x24)
[<c0abc808>] (mutex_lock_nested) from [<c064590c>] (device_del+0x30/0x39c)
[<c064590c>] (device_del) from [<c0645c9c>] (device_unregister+0x24/0x64)

[<c0645c9c>] (device_unregister) from [<c01b10fc>] (srcu_invoke_callbacks+0xcc/0x154)
[<c01b10fc>] (srcu_invoke_callbacks) from [<c01493c4>] (process_one_work+0x234/0x7dc)
[<c01493c4>] (process_one_work) from [<c01499b0>] (worker_thread+0x44/0x51c) [<c01499b0>] (worker_thread) from [<c0150bf4>] (kthread+0x158/0x1a0) [<c0150bf4>] (kthread) from [<c0100114>] (ret_from_fork+0x14/0x20)

Exception stack(0xee907fb0 to 0xee907ff8)

...
kobject_add_internal failed for regulator.28:cpu0 with -EEXIST, don't try to register things with the same name in the same directory.

Best regards
--
Marek Szyprowski, PhD
Samsung R&D Institute Poland

#14 lore raw

Hi Saravana,

Quoted 30 lines · Saravana Kannan
Quoted 8 lines · Marek Szyprowski
Quoted 7 lines · Greg Kroah-Hartman
Quoted 6 lines · Saravana Kannan
Quoted 5 lines · Saravana Kannan
Quoted 3 lines · Greg Kroah-Hartman

Looks semi-sane, but it's too close to the merge window at the moment for me to take this. If there's no objections by the time 5.8-rc1 is out, I'll queue it up in my tree for 5.9-rc1.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line

Another friendly reminder :)

*nudge* *nudge*

Looks sane, given no objections, let's see what linux-next thinks about it...

linux-next is not very happy from this patchset... Starting from next-20200713 I see a few new issues on various Samsung Exynos based boards. Here are examples from Exynos4412-based Odroid U3 board (ARM 32bit, kernel compiled from exynos_defconfig):

Thanks for the bug reports.

Quoted 10 lines
BUG: sleeping function called from invalid context at
kernel/locking/mutex.c:935
in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 12, name: kworker/0:1
2 locks held by kworker/0:1/12:
   #0: ee8074a8 ((wq_completion)rcu_gp){+.+.}-{0:0}, at:
process_one_work+0x174/0x7dc
   #1: ee921f20 ((work_completion)(&sdp->work)){+.+.}-{0:0}, at:
process_one_work+0x174/0x7dc

Preemption disabled at:
[<c01b10f0>] srcu_invoke_callbacks+0xc0/0x154

Quoted 4 lines

Sigh... probably some SRCU screw up when the device link is deleted. I'll look at it by the end of this week. If you don't mind, what SRCU debug config caught this for you? That way, I can reproduce it on my end.

Just the options enabled in the default exynos_defconfig in current linux-next. I didn't check any particular options yet.

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland