driver core: Fix suspend/resume order issue with deferred probe
PATCH 1/1 driver core: Fix suspend/resume order issue with deferred probe
Under the following conditions:
- driver A is built in and can probe device-A
- driver B is a module and can probe device-B
- device-A is supplier of device-B
Without this patch:
1. device-A is added.
2. device-B is added.
3. dpm_list is now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted
6. device-A is moved to end of dpm_list.
6. dpm_list is now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8. dpm_list stays as [device-B, device-A].
Suspend (which goes in the reverse order of dpm_list) fails because device-A (supplier) is suspended before device-B (consumer).
With this patch:
1. device-A is added.
2. device-B is added.
3. dpm_list is now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted later.
6. dpm_list is now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8. dpm_list is now [device-A, device-B].
Suspend works because device-B (consumer) is suspended before device-A (supplier).
--- drivers/base/dd.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
drivers/base/dd.c +16 -0
@@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)* probe makes that very unsafe. */ device_pm_move_to_tail (dev); +/* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but +* I'm worried if it'll have some unintended consequeneces. */ dev_dbg (dev,"Retrying from deferred list\n" );bus_probe_device (dev);
@@ -557,6 +559,20 @@ static int really_probe(struct device *dev, struct device_driver *drv) goto re_probe; } +/* +* The devices are added to the dpm_list (resume/suspend (reverse +* order) list) as they are registered with the driver core. But the +* order the devices are added doesn't necessarily match the real +* dependency order. +* +* The successful probe order is a much better signal. If a device just +* probed successfully, then we know for sure that all the devices that +* probed before it don't depend on the device. So, we can safely move +* the device to the end of the dpm_list. As more devices probe, +* they'll automatically get ordered correctly. +*/ +device_pm_move_to_tail (dev); +pinctrl_init_done (dev);if (dev->pm_domain && dev->pm_domain->sync)
Signature · 1 line
2.27.0.111.gc72c7da667-goog
Hi Saravana,
Thanks for your patch!
Quoted 3 lines · Saravana Kannan
Under the following conditions:
- driver A is built in and can probe device-A
- driver B is a module and can probe device-B
I think this is not correct: in my case driver B is builtin, too.
Quoted 8 lines
- device-A is supplier of device-B
Without this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted
I think this is misleading: in my case driver-A did not defer the probe of device-A, and driver-A never returned -EPROBE_DEFER. Probing was merely paused, due to fw_devlink_pause();
Quoted 18 lines
6. device-A is moved to end of
dpm_list.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_liststays as [device-B, device-A].Suspend (which goes in the reverse order of
dpm_list) fails because device-A (supplier) is suspended before device-B (consumer).With this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted later.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_listis now [device-A, device-B].
Quoted 7 lines
Suspend works because device-B (consumer) is suspended before device-A (supplier).
Fixes 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order")Fixes 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing")Cc Geert Uytterhoeven <[email protected]>Signed-off-by Saravana Kannan <[email protected]>
This fixes wake-up by GPIO key on r8a7740/armadillo and sh73a0/kzm9g.
drivers/base/dd.c static void deferred_probe_work_func(struct work_struct *work) · line 113
Quoted 8 lines
drivers/base/dd.c +2
-0@@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)* probe makes that very unsafe. */ device_pm_move_to_tail (dev); +/* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but +* I'm worried if it'll have some unintended consequeneces. */
Works fine for me with the call to device_pm_move_to_tail() removed, too (at least on the two boards that showed the issue before).
Gr{oetje,eeting}s,
Geert
Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected] In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Dropping Feng and Toan due to mail bounces.
Quoted 11 lines · Geert Uytterhoeven
Hi Saravana,
Thanks for your patch!
Quoted 3 lines · Saravana Kannan
Under the following conditions:
- driver A is built in and can probe device-A
- driver B is a module and can probe device-BI think this is not correct: in my case driver B is builtin, too.
This is a correct example, it just doesn't match with your case :)
Talking about fw_devlink_pause/resume() just distracts from the real issue that's also present in systems that don't use DT.
You have this problem even on an ACPI system -- distributions loading all the modules in a PC. We want suspend/resume to work for those too. So, I'm just going for a simpler example.
Quoted 9 lines
Quoted 8 lines
- device-A is supplier of device-B
Without this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted
Quoted 3 lines
I think this is misleading: in my case driver-A did not defer the probe of device-A, and driver-A never returned -EPROBE_DEFER. Probing was merely paused, due to
fw_devlink_pause();
What I said above. fw_devlink_pause() just defers the probe for the device -- that's how it pauses and resumes probing. For example, device link can defer the probe for a device without ever getting to the driver too.
Quoted 21 lines
Quoted 21 lines
6. device-A is moved to end of
dpm_list.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_liststays as [device-B, device-A].Suspend (which goes in the reverse order of
dpm_list) fails because device-A (supplier) is suspended before device-B (consumer).With this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted later.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_listis now [device-A, device-B].Suspend works because device-B (consumer) is suspended before device-A (supplier).
Quoted 7 lines
Quoted 4 lines
Fixes 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order")Fixes 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing")Cc Geert Uytterhoeven <[email protected]>Signed-off-by Saravana Kannan <[email protected]>This fixes wake-up by GPIO key on r8a7740/armadillo and sh73a0/kzm9g.
Tested-by Geert Uytterhoeven <[email protected]>
Thanks!
Quoted 12 lines
drivers/base/dd.c static void deferred_probe_work_func(struct work_struct *work) · line 113
Quoted 9 lines
drivers/base/dd.c +2
-0@@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)* probe makes that very unsafe. */ device_pm_move_to_tail (dev); +/* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but +* I'm worried if it'll have some unintended consequeneces. */ Works fine for me with the call to
device_pm_move_to_tail()removed, too (at least on the two boards that showed the issue before).
Yes, it feels right to remove this, but I just wanted to get a few more opinions.
-Saravana
Rafael J. Wysocki
maintainerQuoted 46 lines · Saravana Kannan
Under the following conditions:
- driver A is built in and can probe device-A
- driver B is a module and can probe device-B
- device-A is supplier of device-BWithout this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted
6. device-A is moved to end ofdpm_list.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_liststays as [device-B, device-A].Suspend (which goes in the reverse order of
dpm_list) fails because device-A (supplier) is suspended before device-B (consumer).With this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted later.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_listis now [device-A, device-B].Suspend works because device-B (consumer) is suspended before device-A (supplier).
Fixes 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order")Fixes 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing")Cc Geert Uytterhoeven <[email protected]>Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/dd.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/base/dd.c +0
-0
drivers/base/dd.c static void deferred_probe_work_func(struct work_struct *work) · line 113
Quoted 5 lines
drivers/base/dd.c +2
-0@@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)* probe makes that very unsafe. */ device_pm_move_to_tail (dev); +/* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but +* I'm worried if it'll have some unintended consequeneces. */
Yes, this needs to go away if you make the other change.
Quoted 10 lines
drivers/base/dd.c +3
-0dev_dbg (dev,"Retrying from deferred list\n" );bus_probe_device (dev);@@ -557,6 +559,20 @@ static int really_probe(struct device *dev, struct device_driver *drv) goto re_probe; } +/* +* The devices are added to the dpm_list (resume/suspend (reverse +* order) list) as they are registered with the driver core. But the
drivers/base/dd.c static int really_probe(struct device *dev, struct device_driver *drv) · line 574
Quoted 10 lines
drivers/base/dd.c +10
-0+* order the devices are added doesn't necessarily match the real +* dependency order. +* +* The successful probe order is a much better signal. If a device just +* probed successfully, then we know for sure that all the devices that +* probed before it don't depend on the device. So, we can safely move +* the device to the end of the dpm_list. As more devices probe, +* they'll automatically get ordered correctly. +*/ +device_pm_move_to_tail (dev);
But it would be good to somehow limit this to the devices affected by deferred probing or we'll end up reordering dpm_list unnecessarily for many times in the actual majority of cases.
Quoted 7 lines
drivers/base/dd.c +1
-1+pinctrl_init_done (dev);if (dev->pm_domain && dev->pm_domain->sync) --2.27.0.111.gc72c7da667-goog
Dropping Feng Kan <[email protected]> and Toan Le <[email protected]> because their mails are bouncing.
Quoted 72 lines · Rafael J. Wysocki
Quoted 46 lines · Saravana Kannan
Under the following conditions:
- driver A is built in and can probe device-A
- driver B is a module and can probe device-B
- device-A is supplier of device-BWithout this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted
6. device-A is moved to end ofdpm_list.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_liststays as [device-B, device-A].Suspend (which goes in the reverse order of
dpm_list) fails because device-A (supplier) is suspended before device-B (consumer).With this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted later.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_listis now [device-A, device-B].Suspend works because device-B (consumer) is suspended before device-A (supplier).
Fixes 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order")Fixes 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing")Cc Geert Uytterhoeven <[email protected]>Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/dd.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/base/dd.c +0
-0drivers/base/dd.c static void deferred_probe_work_func(struct work_struct *work) · line 113
Quoted 5 lines
drivers/base/dd.c +2
-0@@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)* probe makes that very unsafe. */ device_pm_move_to_tail (dev); +/* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but +* I'm worried if it'll have some unintended consequeneces. */ Yes, this needs to go away if you make the other change.
Quoted 16 lines
drivers/base/dd.c +9
-0dev_dbg (dev,"Retrying from deferred list\n" );bus_probe_device (dev);@@ -557,6 +559,20 @@ static int really_probe(struct device *dev, struct device_driver *drv) goto re_probe; } +/* +* The devices are added to the dpm_list (resume/suspend (reverse +* order) list) as they are registered with the driver core. But the +* order the devices are added doesn't necessarily match the real +* dependency order. +* +* The successful probe order is a much better signal. If a device just +* probed successfully, then we know for sure that all the devices that +* probed before it don't depend on the device. So, we can safely move
Quoted 8 lines
drivers/base/dd.c static int really_probe(struct device *dev, struct device_driver *drv) · line 574
Quoted 4 lines
drivers/base/dd.c +4
-0+* the device to the end of the dpm_list. As more devices probe, +* they'll automatically get ordered correctly. +*/ +device_pm_move_to_tail (dev);But it would be good to somehow limit this to the devices affected by deferred probing or we'll end up reordering
dpm_listunnecessarily for many times in the actual majority of cases.
Yes, lots of unnecessary reordering, but doing it only for deferred probes IS the problem. In the example I gave, the consumer is never deferred probe because the supplier happens to finish probing before the consumer probe is even attempted.
I'm open to other suggestions, but I think this is needed for all the cases or at least more cases to be handled correctly.
One alternative I was thinking was not adding the device to the dpm_list until it's probed. But I have the vague feeling of other things between device_add() and device probe that expect the device to be in the dpm_list.
-Saravana
Rafael J. Wysocki
maintainerQuoted 86 lines · Saravana Kannan
Dropping Feng Kan <[email protected]> and Toan Le <[email protected]> because their mails are bouncing.
Quoted 72 lines · Rafael J. Wysocki
Quoted 46 lines · Saravana Kannan
Under the following conditions:
- driver A is built in and can probe device-A
- driver B is a module and can probe device-B
- device-A is supplier of device-BWithout this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted
6. device-A is moved to end ofdpm_list.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_liststays as [device-B, device-A].Suspend (which goes in the reverse order of
dpm_list) fails because device-A (supplier) is suspended before device-B (consumer).With this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted later.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_listis now [device-A, device-B].Suspend works because device-B (consumer) is suspended before device-A (supplier).
Fixes 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order")Fixes 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing")Cc Geert Uytterhoeven <[email protected]>Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/dd.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/base/dd.c +0
-0drivers/base/dd.c static void deferred_probe_work_func(struct work_struct *work) · line 113
Quoted 5 lines
drivers/base/dd.c +2
-0@@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)* probe makes that very unsafe. */ device_pm_move_to_tail (dev); +/* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but +* I'm worried if it'll have some unintended consequeneces. */ Yes, this needs to go away if you make the other change.
Quoted 16 lines
drivers/base/dd.c +9
-0dev_dbg (dev,"Retrying from deferred list\n" );bus_probe_device (dev);@@ -557,6 +559,20 @@ static int really_probe(struct device *dev, struct device_driver *drv) goto re_probe; } +/* +* The devices are added to the dpm_list (resume/suspend (reverse +* order) list) as they are registered with the driver core. But the +* order the devices are added doesn't necessarily match the real +* dependency order. +* +* The successful probe order is a much better signal. If a device just +* probed successfully, then we know for sure that all the devices that +* probed before it don't depend on the device. So, we can safely move Quoted 8 lines
drivers/base/dd.c static int really_probe(struct device *dev, struct device_driver *drv) · line 574
Quoted 4 lines
drivers/base/dd.c +4
-0+* the device to the end of the dpm_list. As more devices probe, +* they'll automatically get ordered correctly. +*/ +device_pm_move_to_tail (dev);But it would be good to somehow limit this to the devices affected by deferred probing or we'll end up reordering
dpm_listunnecessarily for many times in the actual majority of cases.
Quoted 4 lines
Yes, lots of unnecessary reordering, but doing it only for deferred probes IS the problem. In the example I gave, the consumer is never deferred probe because the supplier happens to finish probing before the consumer probe is even attempted.
But why would the supplier be moved to the end of dpm_list without moving the consumer along with it?
Quoted 88 lines · Rafael J. Wysocki
Quoted 86 lines · Saravana Kannan
Dropping Feng Kan <[email protected]> and Toan Le <[email protected]> because their mails are bouncing.
Quoted 80 lines · Rafael J. Wysocki
Quoted 46 lines · Saravana Kannan
Under the following conditions:
- driver A is built in and can probe device-A
- driver B is a module and can probe device-B
- device-A is supplier of device-BWithout this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted
6. device-A is moved to end ofdpm_list.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_liststays as [device-B, device-A].Suspend (which goes in the reverse order of
dpm_list) fails because device-A (supplier) is suspended before device-B (consumer).With this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted later.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_listis now [device-A, device-B].Suspend works because device-B (consumer) is suspended before device-A (supplier).
Fixes 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order")Fixes 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing")Cc Geert Uytterhoeven <[email protected]>Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/dd.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/base/dd.c +0
-0drivers/base/dd.c static void deferred_probe_work_func(struct work_struct *work) · line 113
Quoted 5 lines
drivers/base/dd.c +2
-0@@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)* probe makes that very unsafe. */ device_pm_move_to_tail (dev); +/* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but +* I'm worried if it'll have some unintended consequeneces. */ Yes, this needs to go away if you make the other change.
Quoted 10 lines
drivers/base/dd.c +3
-0dev_dbg (dev,"Retrying from deferred list\n" );bus_probe_device (dev);@@ -557,6 +559,20 @@ static int really_probe(struct device *dev, struct device_driver *drv) goto re_probe; } +/* +* The devices are added to the dpm_list (resume/suspend (reverse +* order) list) as they are registered with the driver core. But the drivers/base/dd.c static int really_probe(struct device *dev, struct device_driver *drv) · line 574
Quoted 10 lines
drivers/base/dd.c +10
-0+* order the devices are added doesn't necessarily match the real +* dependency order. +* +* The successful probe order is a much better signal. If a device just +* probed successfully, then we know for sure that all the devices that +* probed before it don't depend on the device. So, we can safely move +* the device to the end of the dpm_list. As more devices probe, +* they'll automatically get ordered correctly. +*/ +device_pm_move_to_tail (dev);But it would be good to somehow limit this to the devices affected by deferred probing or we'll end up reordering
dpm_listunnecessarily for many times in the actual majority of cases.
Quoted 7 lines
Quoted 4 lines
Yes, lots of unnecessary reordering, but doing it only for deferred probes IS the problem. In the example I gave, the consumer is never deferred probe because the supplier happens to finish probing before the consumer probe is even attempted.
But why would the supplier be moved to the end of
dpm_listwithout moving the consumer along with it?
There is no device link between the supplier/consumer in this case. Sadly there are plenty of cases where device links aren't present to capture supplier/consumer dependencies.
-Saravana
Rafael J. Wysocki
maintainerQuoted 95 lines · Saravana Kannan
Quoted 93 lines · Rafael J. Wysocki
Quoted 90 lines · Saravana Kannan
Dropping Feng Kan <[email protected]> and Toan Le <[email protected]> because their mails are bouncing.
Quoted 72 lines · Rafael J. Wysocki
Quoted 46 lines · Saravana Kannan
Under the following conditions:
- driver A is built in and can probe device-A
- driver B is a module and can probe device-B
- device-A is supplier of device-BWithout this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted
6. device-A is moved to end ofdpm_list.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_liststays as [device-B, device-A].Suspend (which goes in the reverse order of
dpm_list) fails because device-A (supplier) is suspended before device-B (consumer).With this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted later.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_listis now [device-A, device-B].Suspend works because device-B (consumer) is suspended before device-A (supplier).
Fixes 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order")Fixes 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing")Cc Geert Uytterhoeven <[email protected]>Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/dd.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/base/dd.c +0
-0drivers/base/dd.c static void deferred_probe_work_func(struct work_struct *work) · line 113
Quoted 5 lines
drivers/base/dd.c +2
-0@@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)* probe makes that very unsafe. */ device_pm_move_to_tail (dev); +/* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but +* I'm worried if it'll have some unintended consequeneces. */ Yes, this needs to go away if you make the other change.
Quoted 16 lines
drivers/base/dd.c +9
-0dev_dbg (dev,"Retrying from deferred list\n" );bus_probe_device (dev);@@ -557,6 +559,20 @@ static int really_probe(struct device *dev, struct device_driver *drv) goto re_probe; } +/* +* The devices are added to the dpm_list (resume/suspend (reverse +* order) list) as they are registered with the driver core. But the +* order the devices are added doesn't necessarily match the real +* dependency order. +* +* The successful probe order is a much better signal. If a device just +* probed successfully, then we know for sure that all the devices that +* probed before it don't depend on the device. So, we can safely move Quoted 8 lines
drivers/base/dd.c static int really_probe(struct device *dev, struct device_driver *drv) · line 574
Quoted 4 lines
drivers/base/dd.c +4
-0+* the device to the end of the dpm_list. As more devices probe, +* they'll automatically get ordered correctly. +*/ +device_pm_move_to_tail (dev);But it would be good to somehow limit this to the devices affected by deferred probing or we'll end up reordering
dpm_listunnecessarily for many times in the actual majority of cases.Yes, lots of unnecessary reordering, but doing it only for deferred probes IS the problem. In the example I gave, the consumer is never deferred probe because the supplier happens to finish probing before the consumer probe is even attempted.
Quoted 4 lines
Quoted 2 lines
But why would the supplier be moved to the end of
dpm_listwithout moving the consumer along with it?There is no device link between the supplier/consumer in this case.
So this is the real problem, isn't it?
Quoted 2 lines
Sadly there are plenty of cases where device links aren't present to capture supplier/consumer dependencies.
And so that's why you want to add a ton of overhead to driver probing in all of the cases in which that is not an issue?
Quoted 102 lines · Rafael J. Wysocki
Quoted 95 lines · Saravana Kannan
Quoted 93 lines · Rafael J. Wysocki
Quoted 90 lines · Saravana Kannan
Dropping Feng Kan <[email protected]> and Toan Le <[email protected]> because their mails are bouncing.
Quoted 72 lines · Rafael J. Wysocki
Quoted 46 lines · Saravana Kannan
Under the following conditions:
- driver A is built in and can probe device-A
- driver B is a module and can probe device-B
- device-A is supplier of device-BWithout this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted
6. device-A is moved to end ofdpm_list.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_liststays as [device-B, device-A].Suspend (which goes in the reverse order of
dpm_list) fails because device-A (supplier) is suspended before device-B (consumer).With this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted later.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_listis now [device-A, device-B].Suspend works because device-B (consumer) is suspended before device-A (supplier).
Fixes 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order")Fixes 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing")Cc Geert Uytterhoeven <[email protected]>Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/dd.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/base/dd.c +0
-0drivers/base/dd.c static void deferred_probe_work_func(struct work_struct *work) · line 113
Quoted 5 lines
drivers/base/dd.c +2
-0@@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)* probe makes that very unsafe. */ device_pm_move_to_tail (dev); +/* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but +* I'm worried if it'll have some unintended consequeneces. */ Yes, this needs to go away if you make the other change.
Quoted 16 lines
drivers/base/dd.c +9
-0dev_dbg (dev,"Retrying from deferred list\n" );bus_probe_device (dev);@@ -557,6 +559,20 @@ static int really_probe(struct device *dev, struct device_driver *drv) goto re_probe; } +/* +* The devices are added to the dpm_list (resume/suspend (reverse +* order) list) as they are registered with the driver core. But the +* order the devices are added doesn't necessarily match the real +* dependency order. +* +* The successful probe order is a much better signal. If a device just +* probed successfully, then we know for sure that all the devices that +* probed before it don't depend on the device. So, we can safely move Quoted 8 lines
drivers/base/dd.c static int really_probe(struct device *dev, struct device_driver *drv) · line 574
Quoted 4 lines
drivers/base/dd.c +4
-0+* the device to the end of the dpm_list. As more devices probe, +* they'll automatically get ordered correctly. +*/ +device_pm_move_to_tail (dev);But it would be good to somehow limit this to the devices affected by deferred probing or we'll end up reordering
dpm_listunnecessarily for many times in the actual majority of cases.Yes, lots of unnecessary reordering, but doing it only for deferred probes IS the problem. In the example I gave, the consumer is never deferred probe because the supplier happens to finish probing before the consumer probe is even attempted.
Quoted 4 lines
Quoted 2 lines
But why would the supplier be moved to the end of
dpm_listwithout moving the consumer along with it?There is no device link between the supplier/consumer in this case.
Quoted 7 lines
So this is the real problem, isn't it?
Quoted 2 lines
Sadly there are plenty of cases where device links aren't present to capture supplier/consumer dependencies.
And so that's why you want to add a ton of overhead to driver probing in all of the cases in which that is not an issue?
Well, until all/most of the frameworks add device links or fw_devlink=on by default, it doesn't hurt to have suspend/resume work in more platforms.
What about the option of not adding to dpm_list until a device is probed? Is it DOA? Or can it be made to work?
-Saravana
Rafael J. Wysocki
maintainerQuoted 109 lines · Saravana Kannan
Quoted 107 lines · Rafael J. Wysocki
Quoted 95 lines · Saravana Kannan
Quoted 93 lines · Rafael J. Wysocki
Quoted 90 lines · Saravana Kannan
Dropping Feng Kan <[email protected]> and Toan Le <[email protected]> because their mails are bouncing.
Quoted 72 lines · Rafael J. Wysocki
Quoted 46 lines · Saravana Kannan
Under the following conditions:
- driver A is built in and can probe device-A
- driver B is a module and can probe device-B
- device-A is supplier of device-BWithout this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted
6. device-A is moved to end ofdpm_list.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_liststays as [device-B, device-A].Suspend (which goes in the reverse order of
dpm_list) fails because device-A (supplier) is suspended before device-B (consumer).With this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted later.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_listis now [device-A, device-B].Suspend works because device-B (consumer) is suspended before device-A (supplier).
Fixes 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order")Fixes 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing")Cc Geert Uytterhoeven <[email protected]>Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/dd.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/base/dd.c +0
-0drivers/base/dd.c static void deferred_probe_work_func(struct work_struct *work) · line 113
Quoted 5 lines
drivers/base/dd.c +2
-0@@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)* probe makes that very unsafe. */ device_pm_move_to_tail (dev); +/* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but +* I'm worried if it'll have some unintended consequeneces. */ Yes, this needs to go away if you make the other change.
Quoted 16 lines
drivers/base/dd.c +9
-0dev_dbg (dev,"Retrying from deferred list\n" );bus_probe_device (dev);@@ -557,6 +559,20 @@ static int really_probe(struct device *dev, struct device_driver *drv) goto re_probe; } +/* +* The devices are added to the dpm_list (resume/suspend (reverse +* order) list) as they are registered with the driver core. But the +* order the devices are added doesn't necessarily match the real +* dependency order. +* +* The successful probe order is a much better signal. If a device just +* probed successfully, then we know for sure that all the devices that +* probed before it don't depend on the device. So, we can safely move Quoted 8 lines
drivers/base/dd.c static int really_probe(struct device *dev, struct device_driver *drv) · line 574
Quoted 4 lines
drivers/base/dd.c +4
-0+* the device to the end of the dpm_list. As more devices probe, +* they'll automatically get ordered correctly. +*/ +device_pm_move_to_tail (dev);But it would be good to somehow limit this to the devices affected by deferred probing or we'll end up reordering
dpm_listunnecessarily for many times in the actual majority of cases.Yes, lots of unnecessary reordering, but doing it only for deferred probes IS the problem. In the example I gave, the consumer is never deferred probe because the supplier happens to finish probing before the consumer probe is even attempted.
Quoted 4 lines
Quoted 2 lines
But why would the supplier be moved to the end of
dpm_listwithout moving the consumer along with it?There is no device link between the supplier/consumer in this case.
So this is the real problem, isn't it?
Quoted 2 lines
Sadly there are plenty of cases where device links aren't present to capture supplier/consumer dependencies.
Quoted 6 lines
Quoted 2 lines
And so that's why you want to add a ton of overhead to driver probing in all of the cases in which that is not an issue?
Well, until all/most of the frameworks add device links or fw_devlink=on by default, it doesn't hurt to have suspend/resume work in more platforms.
In the presence of deferred probing, that is.
Note that deferred probing gets in the way here and so the problem is related to it.
Quoted 2 lines
What about the option of not adding to
dpm_listuntil a device is probed? Is it DOA?
Yes, it is, I'm afraid. There are devices without drivers. :-)
Quoted 1 line
Or can it be made to work?
Quoted 114 lines · Rafael J. Wysocki
Quoted 112 lines · Saravana Kannan
Quoted 109 lines · Rafael J. Wysocki
Quoted 95 lines · Saravana Kannan
Quoted 93 lines · Rafael J. Wysocki
Quoted 90 lines · Saravana Kannan
Dropping Feng Kan <[email protected]> and Toan Le <[email protected]> because their mails are bouncing.
Quoted 72 lines · Rafael J. Wysocki
Quoted 46 lines · Saravana Kannan
Under the following conditions:
- driver A is built in and can probe device-A
- driver B is a module and can probe device-B
- device-A is supplier of device-BWithout this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted
6. device-A is moved to end ofdpm_list.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_liststays as [device-B, device-A].Suspend (which goes in the reverse order of
dpm_list) fails because device-A (supplier) is suspended before device-B (consumer).With this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted later.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_listis now [device-A, device-B].Suspend works because device-B (consumer) is suspended before device-A (supplier).
Fixes 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order")Fixes 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing")Cc Geert Uytterhoeven <[email protected]>Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/dd.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/base/dd.c +0
-0drivers/base/dd.c static void deferred_probe_work_func(struct work_struct *work) · line 113
Quoted 5 lines
drivers/base/dd.c +2
-0@@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)* probe makes that very unsafe. */ device_pm_move_to_tail (dev); +/* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but +* I'm worried if it'll have some unintended consequeneces. */ Yes, this needs to go away if you make the other change.
Quoted 16 lines
drivers/base/dd.c +9
-0dev_dbg (dev,"Retrying from deferred list\n" );bus_probe_device (dev);@@ -557,6 +559,20 @@ static int really_probe(struct device *dev, struct device_driver *drv) goto re_probe; } +/* +* The devices are added to the dpm_list (resume/suspend (reverse +* order) list) as they are registered with the driver core. But the +* order the devices are added doesn't necessarily match the real +* dependency order. +* +* The successful probe order is a much better signal. If a device just +* probed successfully, then we know for sure that all the devices that +* probed before it don't depend on the device. So, we can safely move Quoted 8 lines
drivers/base/dd.c static int really_probe(struct device *dev, struct device_driver *drv) · line 574
Quoted 4 lines
drivers/base/dd.c +4
-0+* the device to the end of the dpm_list. As more devices probe, +* they'll automatically get ordered correctly. +*/ +device_pm_move_to_tail (dev);But it would be good to somehow limit this to the devices affected by deferred probing or we'll end up reordering
dpm_listunnecessarily for many times in the actual majority of cases.Yes, lots of unnecessary reordering, but doing it only for deferred probes IS the problem. In the example I gave, the consumer is never deferred probe because the supplier happens to finish probing before the consumer probe is even attempted.
Quoted 4 lines
Quoted 2 lines
But why would the supplier be moved to the end of
dpm_listwithout moving the consumer along with it?There is no device link between the supplier/consumer in this case.
So this is the real problem, isn't it?
Quoted 2 lines
Sadly there are plenty of cases where device links aren't present to capture supplier/consumer dependencies.
And so that's why you want to add a ton of overhead to driver probing in all of the cases in which that is not an issue?
Quoted 8 lines
Quoted 3 lines
Well, until all/most of the frameworks add device links or fw_devlink=on by default, it doesn't hurt to have suspend/resume work in more platforms.
In the presence of deferred probing, that is.
Note that deferred probing gets in the way here and so the problem is related to it.
I mean, we officially support deferred probing. Shouldn't we fix it so that it doesn't break suspend/resume? Also, it's pretty easy to have cases where one module probes multiple device instances and loading it in one order would break dpm_list order for one device and loading it in another order would break it for another device. And there would be no "proper" order to load modules (because module order != device order).
Quoted 4 lines
Quoted 2 lines
What about the option of not adding to
dpm_listuntil a device is probed? Is it DOA?Yes, it is, I'm afraid. There are devices without drivers. :-)
Devices that still suspend/resume without drivers?! I didn't know that was possible.
-Saravana
Rafael J. Wysocki
maintainerQuoted 120 lines · Saravana Kannan
Quoted 118 lines · Rafael J. Wysocki
Quoted 115 lines · Saravana Kannan
Quoted 102 lines · Rafael J. Wysocki
Quoted 95 lines · Saravana Kannan
Quoted 93 lines · Rafael J. Wysocki
Quoted 90 lines · Saravana Kannan
Dropping Feng Kan <[email protected]> and Toan Le <[email protected]> because their mails are bouncing.
Quoted 72 lines · Rafael J. Wysocki
Quoted 46 lines · Saravana Kannan
Under the following conditions:
- driver A is built in and can probe device-A
- driver B is a module and can probe device-B
- device-A is supplier of device-BWithout this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted
6. device-A is moved to end ofdpm_list.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_liststays as [device-B, device-A].Suspend (which goes in the reverse order of
dpm_list) fails because device-A (supplier) is suspended before device-B (consumer).With this patch:
1. device-A is added.
2. device-B is added.
3.dpm_listis now [device-A, device-B].
4. driver-A defers probe of device-A.
5. deferred probe of device-A is reattempted later.
6.dpm_listis now [device-B, device-A].
7. driver-B is loaded and probes device-B.
8.dpm_listis now [device-A, device-B].Suspend works because device-B (consumer) is suspended before device-A (supplier).
Fixes 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order")Fixes 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing")Cc Geert Uytterhoeven <[email protected]>Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/dd.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/base/dd.c +0
-0drivers/base/dd.c static void deferred_probe_work_func(struct work_struct *work) · line 113
Quoted 5 lines
drivers/base/dd.c +2
-0@@ -109,6 +109,8 @@ static void deferred_probe_work_func(struct work_struct *work)* probe makes that very unsafe. */ device_pm_move_to_tail (dev); +/* Greg/Rafael: SHOULD I DELETE THIS? ^^ I think I should, but +* I'm worried if it'll have some unintended consequeneces. */ Yes, this needs to go away if you make the other change.
Quoted 16 lines
drivers/base/dd.c +9
-0dev_dbg (dev,"Retrying from deferred list\n" );bus_probe_device (dev);@@ -557,6 +559,20 @@ static int really_probe(struct device *dev, struct device_driver *drv) goto re_probe; } +/* +* The devices are added to the dpm_list (resume/suspend (reverse +* order) list) as they are registered with the driver core. But the +* order the devices are added doesn't necessarily match the real +* dependency order. +* +* The successful probe order is a much better signal. If a device just +* probed successfully, then we know for sure that all the devices that +* probed before it don't depend on the device. So, we can safely move Quoted 8 lines
drivers/base/dd.c static int really_probe(struct device *dev, struct device_driver *drv) · line 574
Quoted 4 lines
drivers/base/dd.c +4
-0+* the device to the end of the dpm_list. As more devices probe, +* they'll automatically get ordered correctly. +*/ +device_pm_move_to_tail (dev);But it would be good to somehow limit this to the devices affected by deferred probing or we'll end up reordering
dpm_listunnecessarily for many times in the actual majority of cases.Yes, lots of unnecessary reordering, but doing it only for deferred probes IS the problem. In the example I gave, the consumer is never deferred probe because the supplier happens to finish probing before the consumer probe is even attempted.
Quoted 4 lines
Quoted 2 lines
But why would the supplier be moved to the end of
dpm_listwithout moving the consumer along with it?There is no device link between the supplier/consumer in this case.
Quoted 7 lines
So this is the real problem, isn't it?
Quoted 2 lines
Sadly there are plenty of cases where device links aren't present to capture supplier/consumer dependencies.
And so that's why you want to add a ton of overhead to driver probing in all of the cases in which that is not an issue?
Well, until all/most of the frameworks add device links or fw_devlink=on by default, it doesn't hurt to have suspend/resume work in more platforms.
Quoted 7 lines
Quoted 4 lines
In the presence of deferred probing, that is.
Note that deferred probing gets in the way here and so the problem is related to it.
I mean, we officially support deferred probing. Shouldn't we fix it so that it doesn't break suspend/resume?
Yes, we should fix deferred probing.
Quoted 6 lines
Also, it's pretty easy to have
cases where one module probes multiple device instances and loading it in one order would breakdpm_listorder for one device and loading it in another order would break it for another device. And there would be no "proper" order to load modules (because module order != device order).
I'm not saying that the current code is perfect. I'm saying that the fix as proposed adds too much cost for everybody who may not care IMO.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
What about the option of not adding to
dpm_listuntil a device is probed? Is it DOA?Yes, it is, I'm afraid. There are devices without drivers. :-)
Devices that still suspend/resume without drivers?! I didn't know that was possible.
There are "class" devices, "type" devices and devices that simply have no drivers, but the bus type code may still want to touch them during system-wide suspend/resume.
Modules may still not be loaded when the system is suspended for the first time etc.
Quoted 20 lines · Rafael J. Wysocki
Quoted 8 lines · Saravana Kannan
Quoted 3 lines · Rafael J. Wysocki
Note that deferred probing gets in the way here and so the problem is related to it.
I mean, we officially support deferred probing. Shouldn't we fix it so that it doesn't break suspend/resume?
Yes, we should fix deferred probing.
Quoted 6 lines
Also, it's pretty easy to have
cases where one module probes multiple device instances and loading it in one order would breakdpm_listorder for one device and loading it in another order would break it for another device. And there would be no "proper" order to load modules (because module order != device order).
Quoted 2 lines
I'm not saying that the current code is perfect. I'm saying that the fix as proposed adds too much cost for everybody who may not care IMO.
Ok, how about I don't do this reordering until we see the first deferred probe request? Will that work for you? In that case, systems with no deferred probing will not incur any reordering cost. Or if reordering starts only towards the end, all the previous probes won't incur reordering cost.
I'm open to other suggestions too.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Yes, it is, I'm afraid. There are devices without drivers. :-)
Devices that still suspend/resume without drivers?! I didn't know that was possible.
Quoted 6 lines
There are "class" devices, "type" devices and devices that simply have no drivers, but the bus type code may still want to touch them during system-wide suspend/resume.
Modules may still not be loaded when the system is suspended for the first time etc.
Thanks for the context.
-Saravana
Hi Saravana,
Quoted 10 lines · Saravana Kannan
Quoted 9 lines · Rafael J. Wysocki
Quoted 6 lines · Saravana Kannan
Quoted 2 lines · Rafael J. Wysocki
Note that deferred probing gets in the way here and so the problem is related to it.
I mean, we officially support deferred probing. Shouldn't we fix it so that it doesn't break suspend/resume?
Yes, we should fix deferred probing.
Please take into account that breakage is an actual regression.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Also, it's pretty easy to have
cases where one module probes multiple device instances and loading it in one order would breakdpm_listorder for one device and loading it in another order would break it for another device. And there would be no "proper" order to load modules (because module order != device order).
Quoted 8 lines
Quoted 2 lines
I'm not saying that the current code is perfect. I'm saying that the fix as proposed adds too much cost for everybody who may not care IMO.
Ok, how about I don't do this reordering until we see the first deferred probe request? Will that work for you? In that case, systems with no deferred probing will not incur any reordering cost. Or if reordering starts only towards the end, all the previous probes won't incur reordering cost.
That first deferred probe request is more or less as of the first probe, since commit 93d2e4322aa74c1a ("of: platform: Batch fwnode parsing when adding all top level devices"), at least on DT systems.
Gr{oetje,eeting}s,
Geert
Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected] In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Rafael J. Wysocki
maintainerQuoted 33 lines · Geert Uytterhoeven
Hi Saravana,
Quoted 10 lines · Saravana Kannan
Quoted 9 lines · Rafael J. Wysocki
Quoted 6 lines · Saravana Kannan
Quoted 2 lines · Rafael J. Wysocki
Note that deferred probing gets in the way here and so the problem is related to it.
I mean, we officially support deferred probing. Shouldn't we fix it so that it doesn't break suspend/resume?
Yes, we should fix deferred probing.
Please take into account that breakage is an actual regression.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Also, it's pretty easy to have
cases where one module probes multiple device instances and loading it in one order would breakdpm_listorder for one device and loading it in another order would break it for another device. And there would be no "proper" order to load modules (because module order != device order).Quoted 8 lines
Quoted 2 lines
I'm not saying that the current code is perfect. I'm saying that the fix as proposed adds too much cost for everybody who may not care IMO.
Ok, how about I don't do this reordering until we see the first deferred probe request? Will that work for you? In that case, systems with no deferred probing will not incur any reordering cost. Or if reordering starts only towards the end, all the previous probes won't incur reordering cost.
Quoted 3 lines
That first deferred probe request is more or less as of the first probe, since commit
93d2e4322aa74c1a("of: platform: Batch fwnode parsing when adding all top level devices"), at least on DT systems.
The deferred probe reordering of devices to the end of dpm_list started in 2012, so it is nothing new, and it demonstrably works for devices where the dependencies are known to the driver core.
That said, in the cases when the dependencies are known to the driver core, it is also unnecessary to reorder dpm_list in deferred_probe_work_func(), because the right ordering of it is going to be determined elsewhere.
Also commit 494fd7b7ad10 ("PM / core: fix deferred probe breaking suspend resume order") is not the source of the problem here, because the problem would have still been there without it, due to the device_pm_move_last() that was there before, so the Fixes: tag pointing to that commit is misleading.
Now, because 716a7a259690 ("driver core: fw_devlink: Add support for batching fwnode parsing") is an optimization and the regression is present because of it AFAICS, the best way to address it at that point would be to revert commit 716a7a259690 for 5.8 and maybe do the optimization more carefully.
Greg, what do you think?
Greg Kroah-Hartman
maintainerQuoted 54 lines · Rafael J. Wysocki
Quoted 33 lines · Geert Uytterhoeven
Hi Saravana,
Quoted 10 lines · Saravana Kannan
Quoted 9 lines · Rafael J. Wysocki
Quoted 6 lines · Saravana Kannan
Quoted 2 lines · Rafael J. Wysocki
Note that deferred probing gets in the way here and so the problem is related to it.
I mean, we officially support deferred probing. Shouldn't we fix it so that it doesn't break suspend/resume?
Yes, we should fix deferred probing.
Please take into account that breakage is an actual regression.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Also, it's pretty easy to have
cases where one module probes multiple device instances and loading it in one order would breakdpm_listorder for one device and loading it in another order would break it for another device. And there would be no "proper" order to load modules (because module order != device order).Quoted 8 lines
Quoted 2 lines
I'm not saying that the current code is perfect. I'm saying that the fix as proposed adds too much cost for everybody who may not care IMO.
Ok, how about I don't do this reordering until we see the first deferred probe request? Will that work for you? In that case, systems with no deferred probing will not incur any reordering cost. Or if reordering starts only towards the end, all the previous probes won't incur reordering cost.
Quoted 3 lines
That first deferred probe request is more or less as of the first probe, since commit
93d2e4322aa74c1a("of: platform: Batch fwnode parsing when adding all top level devices"), at least on DT systems.The deferred probe reordering of devices to the end of
dpm_liststarted in 2012, so it is nothing new, and it demonstrably works for devices where the dependencies are known to the driver core.That said, in the cases when the dependencies are known to the driver core, it is also unnecessary to reorder
dpm_listindeferred_probe_work_func(), because the right ordering of it is going to be determined elsewhere.Also commit
494fd7b7ad10("PM / core: fix deferred probe breaking suspend resume order") is not the source of the problem here, because the problem would have still been there without it, due to thedevice_pm_move_last()that was there before, so the Fixes: tag pointing to that commit is misleading.
Quoted 7 lines
Now, because
716a7a259690("driver core:fw_devlink: Add support for batching fwnode parsing") is an optimization and the regression is present because of it AFAICS, the best way to address it at that point would be to revert commit716a7a259690for 5.8 and maybe do the optimization more carefully.Greg, what do you think?
I've been ignoreing this and letting you all sort it out :)
But if you think that patch should be reverted, I'll not object and will be glad to to it if this solves the issue.
thanks,
greg k-h
Rafael J. Wysocki
maintainerQuoted 62 lines · Greg Kroah-Hartman
Quoted 60 lines · Rafael J. Wysocki
Quoted 33 lines · Geert Uytterhoeven
Hi Saravana,
Quoted 10 lines · Saravana Kannan
Quoted 9 lines · Rafael J. Wysocki
Quoted 6 lines · Saravana Kannan
Quoted 2 lines · Rafael J. Wysocki
Note that deferred probing gets in the way here and so the problem is related to it.
I mean, we officially support deferred probing. Shouldn't we fix it so that it doesn't break suspend/resume?
Yes, we should fix deferred probing.
Please take into account that breakage is an actual regression.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Also, it's pretty easy to have
cases where one module probes multiple device instances and loading it in one order would breakdpm_listorder for one device and loading it in another order would break it for another device. And there would be no "proper" order to load modules (because module order != device order).Quoted 8 lines
Quoted 2 lines
I'm not saying that the current code is perfect. I'm saying that the fix as proposed adds too much cost for everybody who may not care IMO.
Ok, how about I don't do this reordering until we see the first deferred probe request? Will that work for you? In that case, systems with no deferred probing will not incur any reordering cost. Or if reordering starts only towards the end, all the previous probes won't incur reordering cost.
Quoted 3 lines
That first deferred probe request is more or less as of the first probe, since commit
93d2e4322aa74c1a("of: platform: Batch fwnode parsing when adding all top level devices"), at least on DT systems.The deferred probe reordering of devices to the end of
dpm_liststarted in 2012, so it is nothing new, and it demonstrably works for devices where the dependencies are known to the driver core.That said, in the cases when the dependencies are known to the driver core, it is also unnecessary to reorder
dpm_listindeferred_probe_work_func(), because the right ordering of it is going to be determined elsewhere.Also commit
494fd7b7ad10("PM / core: fix deferred probe breaking suspend resume order") is not the source of the problem here, because the problem would have still been there without it, due to thedevice_pm_move_last()that was there before, so the Fixes: tag pointing to that commit is misleading.Now, because
716a7a259690("driver core:fw_devlink: Add support for batching fwnode parsing") is an optimization and the regression is present because of it AFAICS, the best way to address it at that point would be to revert commit716a7a259690for 5.8 and maybe do the optimization more carefully.
Quoted 6 lines
Quoted 1 line
Greg, what do you think?
I've been ignoreing this and letting you all sort it out :)
But if you think that patch should be reverted, I'll not object and will be glad to to it if this solves the issue.
Well, if Geert can confirm that reverting commit 716a7a259690 makes the problem go away, IMO this would be the most reasonable thing to do at this stage of the cycle without risking that more regressions will be introduced.
Geert?
Quoted 40 lines · Rafael J. Wysocki
Quoted 37 lines · Greg Kroah-Hartman
Quoted 35 lines · Rafael J. Wysocki
Quoted 33 lines · Geert Uytterhoeven
Hi Saravana,
Quoted 10 lines · Saravana Kannan
Quoted 9 lines · Rafael J. Wysocki
Quoted 6 lines · Saravana Kannan
Quoted 2 lines · Rafael J. Wysocki
Note that deferred probing gets in the way here and so the problem is related to it.
I mean, we officially support deferred probing. Shouldn't we fix it so that it doesn't break suspend/resume?
Yes, we should fix deferred probing.
Please take into account that breakage is an actual regression.
Quoted 15 lines
Quoted 9 lines
Quoted 6 lines
Also, it's pretty easy to have
cases where one module probes multiple device instances and loading it in one order would breakdpm_listorder for one device and loading it in another order would break it for another device. And there would be no "proper" order to load modules (because module order != device order).I'm not saying that the current code is perfect. I'm saying that the fix as proposed adds too much cost for everybody who may not care IMO.
Ok, how about I don't do this reordering until we see the first deferred probe request? Will that work for you? In that case, systems with no deferred probing will not incur any reordering cost. Or if reordering starts only towards the end, all the previous probes won't incur reordering cost.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
That first deferred probe request is more or less as of the first probe, since commit
93d2e4322aa74c1a("of: platform: Batch fwnode parsing when adding all top level devices"), at least on DT systems.The deferred probe reordering of devices to the end of
dpm_liststarted in 2012, so it is nothing new, and it demonstrably works for devices where the dependencies are known to the driver core.
Isn't "where the dependencies are known to the driver core" this a big caveat?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
That said, in the cases when the dependencies are known to the driver core, it is also unnecessary to reorder
dpm_listindeferred_probe_work_func(), because the right ordering of it is going to be determined elsewhere.
Until driver core knows about 100% of the dependencies, we still need to do some kind of dpm_list reordering to have correct ordering. Even with fw_devlink=on, I'd imagine it'd be difficult to achieve 100% dependency being known to driver core.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Also commit
494fd7b7ad10("PM / core: fix deferred probe breaking suspend resume order") is not the source of the problem here, because the problem would have still been there without it, due to thedevice_pm_move_last()that was there before, so the Fixes: tag pointing to that commit is misleading.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Now, because
716a7a259690("driver core:fw_devlink: Add support for batching fwnode parsing") is an optimization and the regression is present because of it AFAICS, the best way to address it at that point would be to revert commit716a7a259690for 5.8 and maybe do the optimization more carefully.
No, this patch is not adding any new issues to deferred probe. It just increases the probability of reproducing the issue. That's exactly why I wrote the commit text for this patch without the fwnode batch processing example. Even if you revert the patch, suspend/resume ordering is broken if deferred probe happens.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Greg, what do you think?
I've been ignoreing this and letting you all sort it out :)
Quoted 7 lines
Quoted 2 lines
But if you think that patch should be reverted, I'll not object and will be glad to to it if this solves the issue.
Well, if Geert can confirm that reverting commit
716a7a259690makes the problem go away, IMO this would be the most reasonable thing to do at this stage of the cycle without risking that more regressions will be introduced.
I already have a patch to avoid deferred probe during batch fwnode parsing. I'm trying to do a few more tests before I send it out. So, it'd be nice if we don't revert it right now and give me some time to finish testing.
-Saravana
Rafael J. Wysocki
maintainerQuoted 95 lines · Saravana Kannan
Quoted 40 lines · Rafael J. Wysocki
Quoted 37 lines · Greg Kroah-Hartman
Quoted 35 lines · Rafael J. Wysocki
Quoted 33 lines · Geert Uytterhoeven
Hi Saravana,
Quoted 10 lines · Saravana Kannan
Quoted 9 lines · Rafael J. Wysocki
Quoted 6 lines · Saravana Kannan
Quoted 2 lines · Rafael J. Wysocki
Note that deferred probing gets in the way here and so the problem is related to it.
I mean, we officially support deferred probing. Shouldn't we fix it so that it doesn't break suspend/resume?
Yes, we should fix deferred probing.
Please take into account that breakage is an actual regression.
Quoted 15 lines
Quoted 9 lines
Quoted 6 lines
Also, it's pretty easy to have
cases where one module probes multiple device instances and loading it in one order would breakdpm_listorder for one device and loading it in another order would break it for another device. And there would be no "proper" order to load modules (because module order != device order).I'm not saying that the current code is perfect. I'm saying that the fix as proposed adds too much cost for everybody who may not care IMO.
Ok, how about I don't do this reordering until we see the first deferred probe request? Will that work for you? In that case, systems with no deferred probing will not incur any reordering cost. Or if reordering starts only towards the end, all the previous probes won't incur reordering cost.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
That first deferred probe request is more or less as of the first probe, since commit
93d2e4322aa74c1a("of: platform: Batch fwnode parsing when adding all top level devices"), at least on DT systems.The deferred probe reordering of devices to the end of
dpm_liststarted in 2012, so it is nothing new, and it demonstrably works for devices where the dependencies are known to the driver core.Isn't "where the dependencies are known to the driver core" this a big caveat?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
That said, in the cases when the dependencies are known to the driver core, it is also unnecessary to reorder
dpm_listindeferred_probe_work_func(), because the right ordering of it is going to be determined elsewhere.Until driver core knows about 100% of the dependencies, we still need to do some kind of
dpm_listreordering to have correct ordering. Even with fw_devlink=on, I'd imagine it'd be difficult to achieve 100% dependency being known to driver core.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Also commit
494fd7b7ad10("PM / core: fix deferred probe breaking suspend resume order") is not the source of the problem here, because the problem would have still been there without it, due to thedevice_pm_move_last()that was there before, so the Fixes: tag pointing to that commit is misleading.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Now, because
716a7a259690("driver core:fw_devlink: Add support for batching fwnode parsing") is an optimization and the regression is present because of it AFAICS, the best way to address it at that point would be to revert commit716a7a259690for 5.8 and maybe do the optimization more carefully.No, this patch is not adding any new issues to deferred probe. It just increases the probability of reproducing the issue. That's exactly why I wrote the commit text for this patch without the fwnode batch processing example. Even if you revert the patch, suspend/resume ordering is broken if deferred probe happens.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Greg, what do you think?
I've been ignoreing this and letting you all sort it out :)
Quoted 7 lines
Quoted 2 lines
But if you think that patch should be reverted, I'll not object and will be glad to to it if this solves the issue.
Well, if Geert can confirm that reverting commit
716a7a259690makes the problem go away, IMO this would be the most reasonable thing to do at this stage of the cycle without risking that more regressions will be introduced.
Quoted 4 lines
I already have a patch to avoid deferred probe during batch fwnode parsing. I'm trying to do a few more tests before I send it out. So, it'd be nice if we don't revert it right now and give me some time to finish testing.
If you have an alternative fix, let's see it before deciding what to do.
Greg Kroah-Hartman
maintainerQuoted 94 lines · Saravana Kannan
Quoted 40 lines · Rafael J. Wysocki
Quoted 37 lines · Greg Kroah-Hartman
Quoted 35 lines · Rafael J. Wysocki
Quoted 33 lines · Geert Uytterhoeven
Hi Saravana,
Quoted 10 lines · Saravana Kannan
Quoted 9 lines · Rafael J. Wysocki
Quoted 6 lines · Saravana Kannan
Quoted 2 lines · Rafael J. Wysocki
Note that deferred probing gets in the way here and so the problem is related to it.
I mean, we officially support deferred probing. Shouldn't we fix it so that it doesn't break suspend/resume?
Yes, we should fix deferred probing.
Please take into account that breakage is an actual regression.
Quoted 15 lines
Quoted 9 lines
Quoted 6 lines
Also, it's pretty easy to have
cases where one module probes multiple device instances and loading it in one order would breakdpm_listorder for one device and loading it in another order would break it for another device. And there would be no "proper" order to load modules (because module order != device order).I'm not saying that the current code is perfect. I'm saying that the fix as proposed adds too much cost for everybody who may not care IMO.
Ok, how about I don't do this reordering until we see the first deferred probe request? Will that work for you? In that case, systems with no deferred probing will not incur any reordering cost. Or if reordering starts only towards the end, all the previous probes won't incur reordering cost.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
That first deferred probe request is more or less as of the first probe, since commit
93d2e4322aa74c1a("of: platform: Batch fwnode parsing when adding all top level devices"), at least on DT systems.The deferred probe reordering of devices to the end of
dpm_liststarted in 2012, so it is nothing new, and it demonstrably works for devices where the dependencies are known to the driver core.Isn't "where the dependencies are known to the driver core" this a big caveat?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
That said, in the cases when the dependencies are known to the driver core, it is also unnecessary to reorder
dpm_listindeferred_probe_work_func(), because the right ordering of it is going to be determined elsewhere.Until driver core knows about 100% of the dependencies, we still need to do some kind of
dpm_listreordering to have correct ordering. Even with fw_devlink=on, I'd imagine it'd be difficult to achieve 100% dependency being known to driver core.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Also commit
494fd7b7ad10("PM / core: fix deferred probe breaking suspend resume order") is not the source of the problem here, because the problem would have still been there without it, due to thedevice_pm_move_last()that was there before, so the Fixes: tag pointing to that commit is misleading.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Now, because
716a7a259690("driver core:fw_devlink: Add support for batching fwnode parsing") is an optimization and the regression is present because of it AFAICS, the best way to address it at that point would be to revert commit716a7a259690for 5.8 and maybe do the optimization more carefully.No, this patch is not adding any new issues to deferred probe. It just increases the probability of reproducing the issue. That's exactly why I wrote the commit text for this patch without the fwnode batch processing example. Even if you revert the patch, suspend/resume ordering is broken if deferred probe happens.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Greg, what do you think?
I've been ignoreing this and letting you all sort it out :)
Quoted 7 lines
Quoted 2 lines
But if you think that patch should be reverted, I'll not object and will be glad to to it if this solves the issue.
Well, if Geert can confirm that reverting commit
716a7a259690makes the problem go away, IMO this would be the most reasonable thing to do at this stage of the cycle without risking that more regressions will be introduced.
Quoted 4 lines
I already have a patch to avoid deferred probe during batch fwnode parsing. I'm trying to do a few more tests before I send it out. So, it'd be nice if we don't revert it right now and give me some time to finish testing.
So this series is no longer needed given your other series that I just took?
thanks,
greg k-h
Quoted 9 lines · Greg Kroah-Hartman
Quoted 4 lines · Saravana Kannan
I already have a patch to avoid deferred probe during batch fwnode parsing. I'm trying to do a few more tests before I send it out. So, it'd be nice if we don't revert it right now and give me some time to finish testing.
So this series is no longer needed given your other series that I just took?
This series is no longer needed to fix the issue with fw_devlink optimization that Geert was seeing. The other series you pulled in takes care of Geert's issue.
But deferred probe can still break suspend/resume ordering (example mentioned in the commit text). So I think we should fix that (version X of this patch).
Rafael was concerned about some of the extra work v1 will cause for cases that work fine today. So, we need to find a compromise where we can fix the issue and optimize the fix as much as possible.
One optimization we can do is to call device_pm_move_to_tail(dev) in really_probe() only after deferred probe is triggered (as in the thread gets to run) for the first time. Until then, really_probe() wouldn't have to call device_pm_move_to_tail(dev);
-Saravana
Hi Rafael,
Quoted 67 lines · Rafael J. Wysocki
Quoted 58 lines · Greg Kroah-Hartman
Quoted 57 lines · Rafael J. Wysocki
Quoted 30 lines · Geert Uytterhoeven
Quoted 10 lines · Saravana Kannan
Quoted 9 lines · Rafael J. Wysocki
Quoted 6 lines · Saravana Kannan
Quoted 2 lines · Rafael J. Wysocki
Note that deferred probing gets in the way here and so the problem is related to it.
I mean, we officially support deferred probing. Shouldn't we fix it so that it doesn't break suspend/resume?
Yes, we should fix deferred probing.
Please take into account that breakage is an actual regression.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Also, it's pretty easy to have
cases where one module probes multiple device instances and loading it in one order would breakdpm_listorder for one device and loading it in another order would break it for another device. And there would be no "proper" order to load modules (because module order != device order).Quoted 8 lines
Quoted 2 lines
I'm not saying that the current code is perfect. I'm saying that the fix as proposed adds too much cost for everybody who may not care IMO.
Ok, how about I don't do this reordering until we see the first deferred probe request? Will that work for you? In that case, systems with no deferred probing will not incur any reordering cost. Or if reordering starts only towards the end, all the previous probes won't incur reordering cost.
Quoted 3 lines
That first deferred probe request is more or less as of the first probe, since commit
93d2e4322aa74c1a("of: platform: Batch fwnode parsing when adding all top level devices"), at least on DT systems.The deferred probe reordering of devices to the end of
dpm_liststarted in 2012, so it is nothing new, and it demonstrably works for devices where the dependencies are known to the driver core.That said, in the cases when the dependencies are known to the driver core, it is also unnecessary to reorder
dpm_listindeferred_probe_work_func(), because the right ordering of it is going to be determined elsewhere.Also commit
494fd7b7ad10("PM / core: fix deferred probe breaking suspend resume order") is not the source of the problem here, because the problem would have still been there without it, due to thedevice_pm_move_last()that was there before, so the Fixes: tag pointing to that commit is misleading.Now, because
716a7a259690("driver core:fw_devlink: Add support for batching fwnode parsing") is an optimization and the regression is present because of it AFAICS, the best way to address it at that point would be to revert commit716a7a259690for 5.8 and maybe do the optimization more carefully.Quoted 6 lines
Quoted 1 line
Greg, what do you think?
I've been ignoreing this and letting you all sort it out :)
But if you think that patch should be reverted, I'll not object and will be glad to to it if this solves the issue.
Quoted 6 lines
Well, if Geert can confirm that reverting commit
716a7a259690makes the problem go away, IMO this would be the most reasonable thing to do at this stage of the cycle without risking that more regressions will be introduced.Geert?
Reverting commit 716a7a25969003d8 ("driver core: fw_devlink: Add support for batching fwnode parsing") requires reverting commits fefcfc968723caf9 ("driver core: Remove check in driver_deferred_probe_force_trigger()") and 93d2e4322aa74c1a ("of:
platform: Batch fwnode parsing when adding all top level devices"), too.
While reverting the latter is sufficient to fix the regression for me, I can confirm that reverting all three fixes the issue, too.
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds