shut down devices asynchronously
Subsystems (6)
v22 cover shut down devices asynchronously
These patches are rebased against the driver-core tree's driver-core-next branch and should also apply against recent linux-next. Changes for v18 are contained in patch 3.
This patchset allows the kernel to shutdown devices asynchronously and unrelated async devices to be shut down in parallel to each other.
Only devices which explicitly enable it are shut down asynchronously. The default is for a device to be shut down from the synchronous shutdown loop.
This can dramatically reduce system shutdown/reboot time on systems that have multiple devices that take many seconds to shut down (like certain NVMe drives). On one system tested, the shutdown time went from 11 minutes without this patch to 55 seconds with the patch. And on another system from 80 seconds to 11.
And thank you to everyone who has spent some of their valuable time providing reviews, suggestions, criticisms, or tests on the various iterations of this patchset.
Changes from V21:
- Check a device is alive before accessing its bus pointer
Changes from V20:
- Move PCI bridge async enablement to before its call to device_add
- Use srcu to prevent ensure any active device_add is completed before device_shutdown begins shutting down devices.
Changes from V19:
- Convert initial creation of shutdown_one_device to a code move with no changes or side effects
- Add a warning to device_move should some device ever mix need_parent_lock with device_move in the future
- Link a PCI virtual function as consumer of its associated physical function to ensure proper shutdown order
Changes from V18:
- Fix deadlock and race condition on concurrent device_add
- Use local variable to device_shutdown for caching async_shutdown
- Added dev->p check to avoid NULL pointer deference and tell if the device is not registered so no need to be shutdown. This is to handle the cases when devices consumers list might have devices that are not yet registered.
Changes from V17:
Fix mangled text in kernel parameter description
Re-protect the list removal with the spinlock
* Hold a device reference to ensure the device cannot be freed before attempting list removal
Changes from V16:
Drop spinlock before async subsystem call which uses GFP_KERNEL Handle that async shutdown can widen races between device shutdown and deletion
* __shutdown_one_device will immediately return if a device is dead
* Set shutdown device completion to complete when marking a device dead to prevent waiting on a dead device
* Only late-access a parent pointer if device is in a non-dead state to ensure the pointer is still valid
Changes from V15:
The async_shutdown bit field is converted to a device flags bit Convert all patches to use the flag bit accessor macros to set or check if
async shutdown should be used
Added documentation on the kernel parameter to control use of async shutdown
Changes from V14:
Remove unneeded use of '!!' with boolean type
Changes from V13:
Remove duplicate flagging of async shutdown on scsi hosts/targets/devices
Changes from V12:
Only acquire a parent reference if acquiring the parent's lock device_enable_async_shutdown should return void Minor comment and description cleanups
Changes from V11:
* Swap the order of the first two patches
* Rework conditional parent locking so that lock and unlock no longer use separate conditional checks
* Remove an used variable
* Comment and description text cleanups
Changes from V10:
Reworked to more closely match the design used for async suspend
* No longer uses async subsystem cookies for synchronization
* Minimized changes to struct device
* Enable async shutdown for pci and scsi devices which support async suspend
Changes from V9:
Address resource and timing issues when spawning a unique async thread for every device during shutdown:
* Make the asynchronous threads able to shut down multiple devices, instead of spawning a unique thread for every device.
* Modify core kernel async code with a custom wake function so it doesn't wake up a thread waiting to synchronize on a cookie until the cookie has reached the desired value, instead of waking up every waiting thread to check the cookie every time an async thread ends.
Changes from V8:
Deal with shutdown hangs resulting when a parent/supplier device is
later in the devices_kset list than its children/consumers:
* Ignore sync_state_only devlinks for shutdown dependencies
* Ignore shutdown_after for devices that don't want async shutdown
* Add a sanity check to revert to sync shutdown for any device that would otherwise wait for a child/consumer shutdown that hasn't already been scheduled
Changes from V7:
Do not expose driver async_shutdown_enable in sysfs. Wrapped a long line.
Changes from V6:
Removed a sysfs attribute that allowed the async device shutdown to be "on" (with driver opt-out), "safe" (driver opt-in), or "off"... what was previously "safe" is now the only behavior, so drivers now only need to have the option to enable or disable async shutdown.
Changes from V5:
Separated into multiple patches to make review easier. Reworked some code to make it more readable
Made devices wait for consumers to shut down, not just children
(suggested by David Jeffery)
Changes from V4:
Change code to use cookies for synchronization rather than async domains Allow async shutdown to be disabled via sysfs, and allow driver opt-in or
opt-out of async shutdown (when not disabled), with ability to control driver opt-in/opt-out via sysfs
Changes from V3:
Bug fix (used "parent" not "dev->parent" in device_shutdown)
Changes from V2:
Removed recursive functions to schedule children to be shutdown before
parents, since existing device_shutdown loop will already do this
Changes from V1:
Rewritten using kernel async code (suggested by Lukas Wunner)
David Jeffery (7):
driver core: warn should device_move try to move a need_parent_lock
device
driver core: separate function to shutdown one device driver core: do not always lock parent in shutdown driver core: async device shutdown infrastructure
PCI: Link a virtual function to its physical function
PCI: Enable async shutdown support
scsi: Enable async shutdown support
Tarun Sahu (2):
driver core: rely on put_device to free dev->p driver core: prevent device_add() during system shutdown
.../admin-guide/kernel-parameters.txt | 10 + drivers/base/base.h | 2 + drivers/base/core.c | 269 +++++++++++++++--- drivers/pci/iov.c | 11 +- drivers/pci/probe.c | 2 + drivers/scsi/hosts.c | 2 + drivers/scsi/scsi_sysfs.c | 3 + include/linux/device.h | 2 + 8 files changed, 260 insertions(+), 41 deletions(-)
Signature · 1 line
2.55.0
PATCH v22 1/9 driver core: rely on put_device to free dev->p
From: Tarun Sahu <[email protected]>
device_add allocate private_data for device and assigns to dev->p. If device_add fails in later steps of the function, it cleans up this dev->p which is not necessary because In the next call, put_device free it anyway (if reference to the device is 0 which will be unless someone concurrently get the reference to this device).
This avoids unnecessary races introduced in system. After device is added in device_kset->list by device_add and later steps in the device_add function failures occur, it will free dev->p manually, while in between there might be a user of device_kset->list will take reference to the device just added by device_add. and might try to access dev->p. So relying on put_device to free dev->p prevents such problem.
--- drivers/base/core.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/base/core.c +1 -2
@@ -2618,6 +2618,7 @@ static void device_release(struct kobject *kobj)struct device *dev =kobj_to_dev (kobj);struct device_private *p = dev->p; + dev->p =NULL ;/* * Some platform devices are driven without driver attached * and managed resources may have been acquired. Make sure
@@ -3828,8 +3829,6 @@ int device_add(struct device *dev) parent_error:put_device (parent); name_error: -kfree (dev->p); - dev->p =NULL ; goto done; }EXPORT_SYMBOL_GPL (device_add);
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 1/9] driver core: rely on put_device to free dev->p
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines
From: Tarun Sahu <[email protected]>
device_addallocateprivate_datafor device and assigns todev->p. Ifdevice_addfails in later steps of the function, it cleans up thisdev->pwhich is not necessary because In the next call,put_devicefree it anyway (if reference to the device is 0 which will be unless someone concurrently get the reference to this device).This avoids unnecessary races introduced in system. After device is added in
device_kset->listbydevice_addand later steps in thedevice_addfunction failures occur, it will freedev->pmanually, while in between there might be a user ofdevice_kset->listwill take reference to the device just added bydevice_add. and might try to accessdev->p. So relying onput_deviceto freedev->pprevents such problem.
Quoted 2 lines
Signed-off-by Tarun Sahu <[email protected]>Signed-off-by David Jeffery <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH v22 2/9 driver core: prevent device_add() during system shutdown
In reply to David JefferyFrom: Tarun Sahu <[email protected]>
In Async device shutdown, device_kset->list lock is released to handle asynchronisation and hold again to get entry from device_kset->list. Which will leave window when device_add can try to add the device to device_kset list and temper with ongoing shutdown process. New added device can be async type or sync type and might also introduce new dependency which can cause device_shutdown path to deadlock. S is waiting C to finish but C is never scheduled as it was added recently from device_add path. And C can only be scheduled when main loops continue to reach to C which is waiting on S.
So, When a system enters shutdown (SYSTEM_HALT, SYSTEM_POWER_OFF, or SYSTEM_RESTART), new devices should not be allowed to be added. Adding system_state check (system_is_shutting_down()) to avoid device_add incase of these states of the system. And use srcu so the device shutdown operation can synchronize device_add and ensure any device_add in progress is finished.
--- drivers/base/core.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
drivers/base/core.c +29 -1
@@ -47,6 +47,22 @@ static bool fw_devlink_drv_reg_done;static bool fw_devlink_best_effort;static struct workqueue_struct *device_link_wq; +/** +* system_is_shutting_down - Check if system state is not active. +* +* When system state is not active and in shutdown state, new devices +* should not be allowed to be added. +* +* If system_state is SYSTEM_HALT || SYSTEM_POWER_OFF || SYSTEM_RESTART +* this function will return true. +*/ +static inline bool system_is_shutting_down (void ) +{ +return system_state ==SYSTEM_HALT || + system_state ==SYSTEM_POWER_OFF || + system_state ==SYSTEM_RESTART ; +} +/** * __fwnode_link_add - Create a link between two fwnode_handles. * @con: Consumer end of the link.
@@ -3614,6 +3630,9 @@ static int device_private_init(struct device *dev)return 0 ; } + +DEFINE_STATIC_SRCU (device_add_srcu); +/** * device_add - add device to device hierarchy. * @dev: device.
@@ -3647,13 +3666,20 @@ int device_add(struct device *dev)struct device *parent;struct kobject *kobj;struct class_interface *class_intf; -int error = -EINVAL ; +int idx, error = -EINVAL ;struct kobject *glue_dir =NULL ; + idx =srcu_read_lock (&device_add_srcu); + dev =get_device (dev);if (!dev) goto done; +if (unlikely (system_is_shutting_down ())) { + error = -ESHUTDOWN ; + goto done; + } +if (!dev->p) { error =device_private_init (dev);if (error)
@@ -3803,6 +3829,7 @@ int device_add(struct device *dev) } done:put_device (dev); +srcu_read_unlock (&device_add_srcu, idx);return error; SysEntryError:if (MAJOR (dev->devt))
@@ -4877,6 +4904,7 @@ void device_shutdown(void)wait_for_device_probe ();device_block_probing (); +synchronize_srcu (&device_add_srcu);cpufreq_suspend ();
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 2/9] driver core: prevent device_add() during system shutdown
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 19 lines
From: Tarun Sahu <[email protected]>
In Async device shutdown,
device_kset->listlock is released to handle asynchronisation and hold again to get entry fromdevice_kset->list. Which will leave window whendevice_addcan try to add the device todevice_ksetlist and temper with ongoing shutdown process. New added device can be async type or sync type and might also introduce new dependency which can causedevice_shutdownpath to deadlock. S is waiting C to finish but C is never scheduled as it was added recently fromdevice_addpath. And C can only be scheduled when main loops continue to reach to C which is waiting on S.So, When a system enters shutdown (
SYSTEM_HALT,SYSTEM_POWER_OFF, orSYSTEM_RESTART), new devices should not be allowed to be added. Addingsystem_statecheck (system_is_shutting_down()) to avoiddevice_addincase of these states of the system. And use srcu so the device shutdown operation can synchronizedevice_addand ensure anydevice_addin progress is finished.
Quoted 2 lines
Signed-off-by Tarun Sahu <[email protected]>Signed-off-by David Jeffery <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v22 3/9 driver core: warn should device_move try to move a need_parent_lock device
In reply to David JefferyCurrently, no device has need_parent_lock set and is moved by device_move. need_parent_lock is only set by the usb bus and very few device types ever use device_move.
Add a warning to device_move to catch should it ever be used on a device with need_parent_lock set. The combination would break the immutable relationship needed between parent and child for need_parent_lock when locking and unlocking both.
--- drivers/base/core.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/base/core.c +8 -0
@@ -4705,6 +4705,14 @@ int device_move(struct device *dev, struct device *new_parent,if (!dev)return -EINVAL ; +/* +* device_move() should not be used on devices with need_parent_lock +* set. Concurrent reparenting will violate the immutable +* relationship needed while locking and unlocking both parent and +* child. +*/ +WARN_ON (dev->bus && dev->bus->need_parent_lock); +device_pm_lock (); new_parent =get_device (new_parent); new_parent_kobj =get_device_parent (dev, new_parent);
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 3/9] driver core: warn should device_move try to move a need_parent_lock device
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Currently, no device has
need_parent_lockset and is moved bydevice_move.need_parent_lockis only set by the usb bus and very few device types ever usedevice_move.Add a warning to
device_moveto catch should it ever be used on a device withneed_parent_lockset. The combination would break the immutable relationship needed between parent and child forneed_parent_lockwhen locking and unlocking both.Suggested-by Tarun Sahu <[email protected]>Signed-off-by David Jeffery <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH v22 4/9 driver core: separate function to shutdown one device
In reply to David JefferyMake a separate function for the part of device_shutdown() that does the shutown for a single device. This is in preparation for making device shutdown asynchronous.
--- drivers/base/core.c | 65 ++++++++++++++++++++++++--------------------- 1 file changed, 35 insertions(+), 30 deletions(-)
drivers/base/core.c +35 -30
@@ -4903,6 +4903,40 @@ int device_change_owner(struct device *dev, kuid_t kuid, kgid_t kgid)return error; } +static void shutdown_one_device (struct device *dev,struct device *parent) +{ +/* hold lock to avoid race with probe/release */ +if (parent) +device_lock (parent); +device_lock (dev); + +/* Don't allow any more runtime suspends */ +pm_runtime_get_noresume (dev); +pm_runtime_barrier (dev); + +if (dev->class && dev->class->shutdown_pre) { +if (initcall_debug) +dev_info (dev,"shutdown_pre\n" ); + dev->class->shutdown_pre (dev); + } +if (dev->bus && dev->bus->shutdown) { +if (initcall_debug) +dev_info (dev,"shutdown\n" ); + dev->bus->shutdown (dev); + }else if (dev->driver && dev->driver->shutdown) { +if (initcall_debug) +dev_info (dev,"shutdown\n" ); + dev->driver->shutdown (dev); + } + +device_unlock (dev); +if (parent) +device_unlock (parent); + +put_device (dev); +put_device (parent); +} +/** * device_shutdown - call ->shutdown() on each device to shutdown. */
@@ -4940,36 +4974,7 @@ void device_shutdown(void)list_del_init (&dev->kobj.entry);spin_unlock (&devices_kset->list_lock); -/* hold lock to avoid race with probe/release */ -if (parent) -device_lock (parent); -device_lock (dev); - -/* Don't allow any more runtime suspends */ -pm_runtime_get_noresume (dev); -pm_runtime_barrier (dev); - -if (dev->class && dev->class->shutdown_pre) { -if (initcall_debug) -dev_info (dev,"shutdown_pre\n" ); - dev->class->shutdown_pre (dev); - } -if (dev->bus && dev->bus->shutdown) { -if (initcall_debug) -dev_info (dev,"shutdown\n" ); - dev->bus->shutdown (dev); - }else if (dev->driver && dev->driver->shutdown) { -if (initcall_debug) -dev_info (dev,"shutdown\n" ); - dev->driver->shutdown (dev); - } - -device_unlock (dev); -if (parent) -device_unlock (parent); - -put_device (dev); -put_device (parent); +shutdown_one_device (dev, parent);spin_lock (&devices_kset->list_lock); }
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 4/9] driver core: separate function to shutdown one device
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 6 lines
Make a separate function for the part of
device_shutdown()that does the shutown for a single device. This is in preparation for making device shutdown asynchronous.Signed-off-by Stuart Hayes <[email protected]>Signed-off-by David Jeffery <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
PATCH v22 5/9 driver core: do not always lock parent in shutdown
In reply to David JefferyDon't lock a parent device unless it is needed in device_shutdown. This is in preparation for making device shutdown asynchronous, when it will be needed to allow children of a common parent to shut down simultaneously.
And only acquire a reference to the parent device if the parent is to be locked.
--- drivers/base/core.c | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-)
drivers/base/core.c +28 -16
@@ -4903,12 +4903,10 @@ int device_change_owner(struct device *dev, kuid_t kuid, kgid_t kgid)return error; } -static void shutdown_one_device (struct device *dev,struct device *parent) +static void __shutdown_one_device (struct device *dev) { -/* hold lock to avoid race with probe/release */ -if (parent) -device_lock (parent); -device_lock (dev); +if (!dev->p || dev->p->dead) +return ;/* Don't allow any more runtime suspends */ pm_runtime_get_noresume (dev);
@@ -4928,13 +4926,33 @@ static void shutdown_one_device(struct device *dev, struct device *parent)dev_info (dev,"shutdown\n" ); dev->driver->shutdown (dev); } +} -device_unlock (dev); -if (parent) +static void shutdown_one_device (struct device *dev) +{ +struct device *parent; + +device_lock (dev); + +/* use parent lock if needed to avoid race with probe/release */ +if (dev->p && !dev->p->dead && dev->bus && dev->bus->need_parent_lock && + (parent =get_device (dev->parent))) { +/* the parent lock needs to be acquired first, so re-lock */ +device_unlock (dev); + +device_lock (parent); +device_lock (dev); + +__shutdown_one_device (dev); +device_unlock (dev);device_unlock (parent); +put_device (parent); + }else { +__shutdown_one_device (dev); +device_unlock (dev); + }put_device (dev); -put_device (parent); }/**
@@ -4942,7 +4960,7 @@ static void shutdown_one_device(struct device *dev, struct device *parent)*/ void device_shutdown (void ) { -struct device *dev, *parent; +struct device *dev;wait_for_device_probe ();device_block_probing ();
@@ -4960,12 +4978,6 @@ void device_shutdown(void) dev = list_entry(devices_kset->list.prev, struct device, kobj.entry); -/* -* hold reference count of device's parent to -* prevent it from being freed because parent's -* lock is to be held -*/ - parent =get_device (dev->parent);get_device (dev);/* * Make sure the device is off the kset list, in the
@@ -4974,7 +4986,7 @@ void device_shutdown(void)list_del_init (&dev->kobj.entry);spin_unlock (&devices_kset->list_lock); -shutdown_one_device (dev, parent); +shutdown_one_device (dev);spin_lock (&devices_kset->list_lock); }
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 5/9] driver core: do not always lock parent in shutdown
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Don't lock a parent device unless it is needed in
device_shutdown. This is in preparation for making device shutdown asynchronous, when it will be needed to allow children of a common parent to shut down simultaneously.And only acquire a reference to the parent device if the parent is to be locked.
Signed-off-by Stuart Hayes <[email protected]>Signed-off-by David Jeffery <[email protected]>Signed-off-by Tarun Sahu <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
PATCH v22 6/9 driver core: async device shutdown infrastructure
In reply to David JefferyPatterned after async suspend, allow devices to mark themselves as wanting to perform async shutdown. Devices using async shutdown wait only for their dependencies to shutdown before executing their shutdown routine.
Sync shutdown devices are shut down one at a time and will only wait for an async shutdown device if the async device is a dependency.
Enabled by default, async shutdown can be explicitly enabled or disabled by using the kernel parameter "core.async_shutdown=<bool>"
--- .../admin-guide/kernel-parameters.txt | 10 ++ drivers/base/base.h | 2 + drivers/base/core.c | 141 +++++++++++++++++- include/linux/device.h | 2 + 4 files changed, 154 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +10 -0
@@ -995,6 +995,16 @@ Kernel parameters seconds. A value of 0 disables the blank timer. Defaults to 0. + core.async_shutdown= + [KNL] + Format: <bool> + Enable or disable asynchronous shutdown support. When + enabled, on system shutdown unrelated devices flagged + as async shutdown compatible may be shut down in + parallel and asynchronously. When disabled, device + shutdown is performed serially and synchronously. + Enabled by default. + coredump_filter= [KNL] Change the default value for /proc/<pid>/coredump_filter.
drivers/base/base.h +2 -0
@@ -103,6 +103,7 @@ struct driver_private {* dev_err_probe() for later retrieval via debugfs * @device: pointer back to the struct device that this structure is * associated with. +* @complete: completion for device shutdown ordering * @dead: This device is currently either in the process of or has been * removed from the system. Any asynchronous events scheduled for this * device should exit without taking any action.
@@ -119,6 +120,7 @@ struct device_private {const struct device_driver *async_driver;char *deferred_probe_reason;struct device *device; +struct completion complete; u8 dead:1; };#define to_device_private_parent (obj) \
drivers/base/core.c +140 -1
199 lines not on this page · raw message
include/linux/device.h +2 -0
@@ -612,6 +612,7 @@ enum struct_device_flags {DEV_FLAG_OF_NODE_REUSED =7 ,DEV_FLAG_OFFLINE_DISABLED =8 ,DEV_FLAG_OFFLINE =9 , +DEV_FLAG_ASYNC_SHUTDOWN =10 , DEV_FLAG_COUNT };
@@ -829,6 +830,7 @@ __create_dev_flag_accessors(dma_coherent, DEV_FLAG_DMA_COHERENT);__create_dev_flag_accessors (of_node_reused,DEV_FLAG_OF_NODE_REUSED );__create_dev_flag_accessors (offline_disabled,DEV_FLAG_OFFLINE_DISABLED );__create_dev_flag_accessors (offline,DEV_FLAG_OFFLINE ); +__create_dev_flag_accessors (async_shutdown,DEV_FLAG_ASYNC_SHUTDOWN );#undef __create_dev_flag_accessors
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 6/9] driver core: async device shutdown infrastructure
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines
Patterned after async suspend, allow devices to mark themselves as wanting to perform async shutdown. Devices using async shutdown wait only for their dependencies to shutdown before executing their shutdown routine.
Sync shutdown devices are shut down one at a time and will only wait for an async shutdown device if the async device is a dependency.
Quoted 6 lines
Enabled by default, async shutdown can be explicitly enabled or disabled by using the kernel parameter "core.async_shutdown=<bool>"
Signed-off-by David Jeffery <[email protected]>Signed-off-by Stuart Hayes <[email protected]>Signed-off-by Tarun Sahu <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
PATCH v22 7/9 PCI: Link a virtual function to its physical function
In reply to David JefferyWith a PCI virtual function having the same bus for its parent as the physical function it is associated with, there is no explicit dependency between the two in the device tree. With async shutdown, this can result in the PF being shutdown before the VF has an opportunity to shutdown.
Link the VF as a consumer of the PF to note its dependency and ensure clean shutdown ordering.
--- drivers/pci/iov.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/pci/iov.c +9 -2
@@ -382,14 +382,20 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id) }pci_device_add (virtfn, virtfn->bus); +if (!device_link_add (&virtfn->dev, &dev->dev,DL_FLAG_STATELESS )) { + rc = -ENOMEM ; + goto failed1; + } + rc =pci_iov_sysfs_link (dev, virtfn, id);if (rc) - goto failed1; + goto unlink;pci_bus_add_device (virtfn);return 0 ; - +unlink: +device_link_remove (&virtfn->dev, &dev->dev); failed1:pci_stop_and_remove_bus_device (virtfn);pci_dev_put (dev);
@@ -421,6 +427,7 @@ void pci_iov_remove_virtfn(struct pci_dev *dev, int id)if (virtfn->dev.kobj.sd)sysfs_remove_link (&virtfn->dev.kobj,"physfn" ); +device_link_remove (&virtfn->dev, &dev->dev);pci_stop_and_remove_bus_device (virtfn);virtfn_remove_bus (dev->bus, virtfn->bus);
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 7/9] PCI: Link a virtual function to its physical function
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
With a PCI virtual function having the same bus for its parent as the physical function it is associated with, there is no explicit dependency between the two in the device tree. With async shutdown, this can result in the PF being shutdown before the VF has an opportunity to shutdown.
Link the VF as a consumer of the PF to note its dependency and ensure clean shutdown ordering.
Signed-off-by Tarun Sahu <[email protected]>Signed-off-by David Jeffery <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
PATCH v22 8/9 PCI: Enable async shutdown support
In reply to David JefferyLike its async suspend support, allow PCI device shutdown to be performed asynchronously to reduce shutdown time.
--- drivers/pci/probe.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/pci/probe.c +2 -0
@@ -1038,6 +1038,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge)if (err) goto free; +dev_set_async_shutdown (&bridge->dev);/* Temporarily move resources off the list */ list_splice_init (&bridge->windows, &resources); err =device_add (&bridge->dev);
@@ -2748,6 +2749,7 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)pci_reassigndev_resource_alignment (dev);pci_init_capabilities (dev); +dev_set_async_shutdown (&dev->dev);platform_pci_configure_wake (dev);
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 8/9] PCI: Enable async shutdown support
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 5 lines
Like its async suspend support, allow PCI device shutdown to be performed asynchronously to reduce shutdown time.
Signed-off-by David Jeffery <[email protected]>Signed-off-by Stuart Hayes <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
PATCH v22 9/9 scsi: Enable async shutdown support
In reply to David JefferyLike scsi's async suspend support, allow scsi devices to be shut down asynchronously to reduce system shutdown time.
--- drivers/scsi/hosts.c | 2 ++ drivers/scsi/scsi_sysfs.c | 3 +++ 2 files changed, 5 insertions(+)
drivers/scsi/hosts.c +2 -0
@@ -272,6 +272,7 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,pm_runtime_set_active (&shost->shost_gendev);pm_runtime_enable (&shost->shost_gendev);device_enable_async_suspend (&shost->shost_gendev); +dev_set_async_shutdown (&shost->shost_gendev); error =device_add (&shost->shost_gendev);if (error)
@@ -282,6 +283,7 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,get_device (shost->shost_gendev.parent);device_enable_async_suspend (&shost->shost_dev); +dev_set_async_shutdown (&shost->shost_dev);get_device (&shost->shost_gendev); error =device_add (&shost->shost_dev);
drivers/scsi/scsi_sysfs.c +3 -0
@@ -1387,6 +1387,7 @@ static int scsi_target_add(struct scsi_target *starget)pm_runtime_set_active (&starget->dev);pm_runtime_enable (&starget->dev);device_enable_async_suspend (&starget->dev); +dev_set_async_shutdown (&starget->dev);return 0 ; }
@@ -1413,6 +1414,7 @@ int scsi_sysfs_add_sdev(struct scsi_device *sdev)transport_configure_device (&starget->dev);device_enable_async_suspend (&sdev->sdev_gendev); +dev_set_async_shutdown (&sdev->sdev_gendev);scsi_autopm_get_target (starget);pm_runtime_set_active (&sdev->sdev_gendev);if (!sdev->rpm_autosuspend)
@@ -1432,6 +1434,7 @@ int scsi_sysfs_add_sdev(struct scsi_device *sdev) }device_enable_async_suspend (&sdev->sdev_dev); +dev_set_async_shutdown (&sdev->sdev_dev); error =device_add (&sdev->sdev_dev);if (error) { sdev_printk(KERN_INFO, sdev,
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 9/9] scsi: Enable async shutdown support
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 5 lines
Like scsi's async suspend support, allow scsi devices to be shut down asynchronously to reduce system shutdown time.
Signed-off-by David Jeffery <[email protected]>Signed-off-by Stuart Hayes <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Quoted 17 lines · David Jeffery
These patches are rebased against the driver-core tree's driver-core-next branch and should also apply against recent linux-next. Changes for v18 are contained in patch 3.
This patchset allows the kernel to shutdown devices asynchronously and unrelated async devices to be shut down in parallel to each other.
Only devices which explicitly enable it are shut down asynchronously. The default is for a device to be shut down from the synchronous shutdown loop.
This can dramatically reduce system shutdown/reboot time on systems that have multiple devices that take many seconds to shut down (like certain NVMe drives). On one system tested, the shutdown time went from 11 minutes without this patch to 55 seconds with the patch. And on another system from 80 seconds to 11.
Quoted 7 lines
And thank you to everyone who has spent some of their valuable time providing reviews, suggestions, criticisms, or tests on the various iterations of this patchset.
Changes from V21:
- Check a device is alive before accessing its bus pointer
And this has resolved Sashiko's last complaint. Sashiko found no issues with the v22 patchset.
David Jeffery