Skip to contentSkip to conversation

r8152: use GFP_NOIO during system suspend

PATCH In review Chih Kai Hsu 2 participants 6 messages Aug 10 – Aug 24

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 r8152: use GFP_NOIO during system suspend

During system suspend, memory allocation with GFP_KERNEL can block waiting for I/O to complete. If that I/O depends on a device that is itself suspended, a deadlock results.

Introduce RTL8152_SYSTEM_SUSPEND flag to track when the driver is operating in the system suspend/resume context. Set the flag at the start of rtl8152_system_suspend() and clear it at the end of rtl8152_system_resume(), following the same pattern used by SELECTIVE_SUSPEND for runtime suspend.

In get_registers() and set_registers(), select GFP_NOIO when the flag is set so that the kmalloc and kmemdup calls in those paths do not trigger I/O reclaim.

---
 drivers/net/usb/r8152.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/net/usb/r8152.c +15 -2
index f61686433031..57d04af6368c 100644
@@ -792,6 +792,7 @@ enum rtl8152_flags {
 	IN_PRE_RESET,
 	PROBED_WITH_NO_ERRORS,
 	PROBE_SHOULD_RETRY,
+	RTL8152_SYSTEM_SUSPEND,
 };
 
 #define DEVICE_ID_LENOVO_USB_C_TRAVEL_HUB		0x721e
@@ -1370,9 +1371,12 @@ static
 int get_registers(struct r8152 *tp, u16 value, u16 index, u16 size, void *data)
 {
 	int ret;
+	gfp_t gfp;
 	void *tmp;
 
-	tmp = kmalloc(size, GFP_KERNEL);
+	gfp = test_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags) ? GFP_NOIO :
+							     GFP_KERNEL;
+	tmp = kmalloc(size, gfp);
 	if (!tmp)
 		return -ENOMEM;
 
@@ -1394,9 +1398,12 @@ static
 int set_registers(struct r8152 *tp, u16 value, u16 index, u16 size, void *data)
 {
 	int ret;
+	gfp_t gfp;
 	void *tmp;
 
-	tmp = kmemdup(data, size, GFP_KERNEL);
+	gfp = test_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags) ? GFP_NOIO :
+							     GFP_KERNEL;
+	tmp = kmemdup(data, size, gfp);
 	if (!tmp)
 		return -ENOMEM;
 
@@ -8693,6 +8700,9 @@ static int rtl8152_system_resume(struct r8152 *tp)
 		usb_submit_urb(tp->intr_urb, GFP_NOIO);
 	}
 
+	clear_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags);
+	smp_mb__after_atomic();
+
 	return 0;
 }
 
@@ -8758,6 +8768,9 @@ static int rtl8152_system_suspend(struct r8152 *tp)
 {
 	struct net_device *netdev = tp->netdev;
 
+	set_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags);
+	smp_mb__after_atomic();
+
 	netif_device_detach(netdev);
 
 	if (netif_running(netdev) && test_bit(WORK_ENABLE, &tp->flags)) {
Signature · 1 line
2.34.1
#1 lore raw
Quoted 10 lines · Chih Kai Hsu

During system suspend, memory allocation with GFP_KERNEL can block waiting for I/O to complete. If that I/O depends on a device that is itself suspended, a deadlock results.

Introduce RTL8152_SYSTEM_SUSPEND flag to track when the driver is operating in the system suspend/resume context. Set the flag at the start of rtl8152_system_suspend() and clear it at the end of rtl8152_system_resume(), following the same pattern used by SELECTIVE_SUSPEND for runtime suspend.

Quoted 3 lines

In get_registers() and set_registers(), select GFP_NOIO when the flag is set so that the kmalloc and kmemdup calls in those paths do not trigger I/O reclaim.

Do you have a reproducer for this? Or is it theoretical?

Quoted 11 lines
drivers/net/usb/r8152.c +3 -0
@@ -8693,6 +8700,9 @@ static int rtl8152_system_resume(struct r8152 *tp)
 		usb_submit_urb(tp->intr_urb, GFP_NOIO);
 	}
 
+	clear_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags);
+	smp_mb__after_atomic();
+
 	return 0;
 }
 

drivers/net/usb/r8152.c static int rtl8152_system_suspend(struct r8152 *tp) · line 8776

Quoted 9 lines
drivers/net/usb/r8152.c +3 -0
@@ -8758,6 +8768,9 @@ static int rtl8152_system_suspend(struct r8152 *tp)
 {
 	struct net_device *netdev = tp->netdev;
 
+	set_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags);
+	smp_mb__after_atomic();
+
 	netif_device_detach(netdev);
 
 	if (netif_running(netdev) && test_bit(WORK_ENABLE, &tp->flags)) {

It seems odd to me that every driver needs to track if it is in suspend/resume or not. Doesn't the struct device tell you? Is there no global state somewhere?

Andrew

#2 lore raw
Quoted 11 lines · Andrew Lunn
Quoted 10 lines · Chih Kai Hsu

During system suspend, memory allocation with GFP_KERNEL can block waiting for I/O to complete. If that I/O depends on a device that is itself suspended, a deadlock results.

Introduce RTL8152_SYSTEM_SUSPEND flag to track when the driver is operating in the system suspend/resume context. Set the flag at the start of rtl8152_system_suspend() and clear it at the end of rtl8152_system_resume(), following the same pattern used by SELECTIVE_SUSPEND for runtime suspend.

Quoted 6 lines
Quoted 3 lines

In get_registers() and set_registers(), select GFP_NOIO when the flag is set so that the kmalloc and kmemdup calls in those paths do not trigger I/O reclaim.

Do you have a reproducer for this? Or is it theoretical?

It is theoretical.

Quoted 19 lines
Quoted 19 lines
drivers/net/usb/r8152.c +6 -0
@@ -8693,6 +8700,9 @@ static int rtl8152_system_resume(struct r8152 *tp)
              usb_submit_urb(tp->intr_urb, GFP_NOIO);
      }
 
+     clear_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags);
+     smp_mb__after_atomic();
+
      return 0;
 }
 
@@ -8758,6 +8768,9 @@ static int rtl8152_system_suspend(struct r8152
 *tp)  {
      struct net_device *netdev = tp->netdev;
 
+     set_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags);
+     smp_mb__after_atomic();
+
      netif_device_detach(netdev);
 
Quoted 5 lines
Quoted 2 lines
(quoted diff) +0 -0
      if (netif_running(netdev) && test_bit(WORK_ENABLE, &tp->flags))

{

It seems odd to me that every driver needs to track if it is in suspend/resume or not. Doesn't the struct device tell you? Is there no global state somewhere?

I found https://elixir.bootlin.com/linux/v7.2-rc6/source/drivers/base/power/main.c#L2343
in the Linux source code.

pm_restrict_gfp_mask() is called before dpm_suspend(state). It removes __GFP_IO and __GFP_FS from the global gfp_t gfp_allowed_mask. Therefore, I think the driver can use GFP_KERNEL safely. Is this correct? Could you please provide any advice?

Quoted 2 lines

Andrew

Best,
Chih-Kai

#3 lore raw
Quoted 60 lines · Andrew Lunn
Quoted 11 lines · Chih Kai Hsu
Quoted 10 lines

During system suspend, memory allocation with GFP_KERNEL can block waiting for I/O to complete. If that I/O depends on a device that is itself suspended, a deadlock results.

Introduce RTL8152_SYSTEM_SUSPEND flag to track when the driver is operating in the system suspend/resume context. Set the flag at the start of rtl8152_system_suspend() and clear it at the end of rtl8152_system_resume(), following the same pattern used by SELECTIVE_SUSPEND for runtime suspend.

Quoted 6 lines
Quoted 3 lines

In get_registers() and set_registers(), select GFP_NOIO when the flag is set so that the kmalloc and kmemdup calls in those paths do not trigger I/O reclaim.

Do you have a reproducer for this? Or is it theoretical?

It is theoretical.

Quoted 1 line

Review static int rtl8152_system_resume(struct r8152

Quoted 1 line
(quoted diff) +0 -0
@@ -8693,6 +8700,9 @@ static int rtl8152_system_resume(struct r8152

*tp)

Quoted 18 lines
Quoted 18 lines
drivers/net/usb/r8152.c +6 -0
              usb_submit_urb(tp->intr_urb, GFP_NOIO);
      }
 
+     clear_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags);
+     smp_mb__after_atomic();
+
      return 0;
 }
 
@@ -8758,6 +8768,9 @@ static int rtl8152_system_suspend(struct r8152
 *tp)  {
      struct net_device *netdev = tp->netdev;
 
+     set_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags);
+     smp_mb__after_atomic();
+
      netif_device_detach(netdev);
 
Quoted 5 lines
Quoted 2 lines
(quoted diff) +0 -0
      if (netif_running(netdev) && test_bit(WORK_ENABLE,

&tp->flags)) {

It seems odd to me that every driver needs to track if it is in suspend/resume or not. Doesn't the struct device tell you? Is there no global

state somewhere?

I found
https://elixir.bootlin.com/linux/v7.2-rc6/source/drivers/base/power/main.c#L
2343
in the Linux source code.

pm_restrict_gfp_mask() is called before dpm_suspend(state). It removes __GFP_IO and __GFP_FS from the global gfp_t gfp_allowed_mask. Therefore, I think the driver can use GFP_KERNEL safely. Is this correct? Could you please provide any advice?

Quoted 1 line
Quoted 4 lines
Quoted 1 line

Andrew

Best,
Chih-Kai

Hi Andrew,

Just a gentle ping on this patch.
Based on my previous observation, should the driver use GFP_NOIO explicitly?

Thank you for your time.

Best,
Chih-Kai

#4 lore raw
Quoted 62 lines · Chih Kai Hsu
Quoted 62 lines · Andrew Lunn
Quoted 11 lines · Chih Kai Hsu
Quoted 10 lines

During system suspend, memory allocation with GFP_KERNEL can block waiting for I/O to complete. If that I/O depends on a device that is itself suspended, a deadlock results.

Introduce RTL8152_SYSTEM_SUSPEND flag to track when the driver is operating in the system suspend/resume context. Set the flag at the start of rtl8152_system_suspend() and clear it at the end of rtl8152_system_resume(), following the same pattern used by SELECTIVE_SUSPEND for runtime suspend.

Quoted 6 lines
Quoted 3 lines

In get_registers() and set_registers(), select GFP_NOIO when the flag is set so that the kmalloc and kmemdup calls in those paths do not trigger I/O reclaim.

Do you have a reproducer for this? Or is it theoretical?

It is theoretical.

Quoted 1 line

Review static int rtl8152_system_resume(struct r8152

Quoted 1 line
(quoted diff) +0 -0
@@ -8693,6 +8700,9 @@ static int rtl8152_system_resume(struct r8152

*tp)

Quoted 18 lines
Quoted 18 lines
drivers/net/usb/r8152.c +6 -0
              usb_submit_urb(tp->intr_urb, GFP_NOIO);
      }
 
+     clear_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags);
+     smp_mb__after_atomic();
+
      return 0;
 }
 
@@ -8758,6 +8768,9 @@ static int rtl8152_system_suspend(struct r8152
 *tp)  {
      struct net_device *netdev = tp->netdev;
 
+     set_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags);
+     smp_mb__after_atomic();
+
      netif_device_detach(netdev);
 
Quoted 5 lines
Quoted 2 lines
(quoted diff) +0 -0
      if (netif_running(netdev) && test_bit(WORK_ENABLE,

&tp->flags)) {

It seems odd to me that every driver needs to track if it is in suspend/resume or not. Doesn't the struct device tell you? Is there no global

state somewhere?

I found
https://elixir.bootlin.com/linux/v7.2-rc6/source/drivers/base/power/main.c#L
2343
in the Linux source code.

pm_restrict_gfp_mask() is called before dpm_suspend(state). It removes __GFP_IO and __GFP_FS from the global gfp_t gfp_allowed_mask. Therefore, I think the driver can use GFP_KERNEL safely. Is this correct? Could you please provide any advice?

Quoted 2 lines

Andrew

Quoted 7 lines
Quoted 2 lines

Best,
Chih-Kai

Hi Andrew,

Just a gentle ping on this patch.
Based on my previous observation, should the driver use GFP_NOIO explicitly?

Please try to answer the question yourself. There are lots of drivers out there. What do they do? What does the suspend/resume documentation say?

Andrew

#5 lore raw
Quoted 44 lines · Andrew
Quoted 43 lines · Chih Kai Hsu
Quoted 43 lines · Andrew Lunn
Quoted 11 lines · Chih Kai Hsu
Quoted 10 lines

During system suspend, memory allocation with GFP_KERNEL can block waiting for I/O to complete. If that I/O depends on a device that is itself suspended, a deadlock results.

Introduce RTL8152_SYSTEM_SUSPEND flag to track when the driver is operating in the system suspend/resume context. Set the flag at the start of rtl8152_system_suspend() and clear it at the end of rtl8152_system_resume(), following the same pattern used by SELECTIVE_SUSPEND for runtime suspend.

Quoted 6 lines
Quoted 3 lines

In get_registers() and set_registers(), select GFP_NOIO when the flag is set so that the kmalloc and kmemdup calls in those paths do not trigger I/O reclaim.

Do you have a reproducer for this? Or is it theoretical?

It is theoretical.

Quoted 2 lines
Quoted 2 lines
(quoted diff) +0 -0
@@ -8693,6 +8700,9 @@ static int rtl8152_system_resume(struct

r8152

*tp)

Quoted 19 lines
Quoted 19 lines
              usb_submit_urb(tp->intr_urb, GFP_NOIO);
      }
drivers/net/usb/r8152.c +3 -0
+     clear_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags);
+     smp_mb__after_atomic();
+
      return 0;
 }
@@ -8758,6 +8768,9 @@ static int rtl8152_system_suspend(struct

r8152

*tp)  {
      struct net_device *netdev = tp->netdev;
drivers/net/usb/r8152.c +3 -0
+     set_bit(RTL8152_SYSTEM_SUSPEND, &tp->flags);
+     smp_mb__after_atomic();
+
      netif_device_detach(netdev);
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
      if (netif_running(netdev) && test_bit(WORK_ENABLE,

&tp->flags)) {

It seems odd to me that every driver needs to track if it is in suspend/resume or not. Doesn't the struct device tell you? Is there no global

state somewhere?

The function hibernate_quiet_exec() used by the driver at drivers/nvdimm/core.c calls dpm_suspend() directly, so the driver cannot use functions such as pm_suspend_in_progress() or system_entering_hibernation() to determine whether it is in the suspend/hibernate path now.

The ax88179 driver at drivers/net/usb/ax88179_178a.c uses a private variable, in_pm, which is set when entering the suspend callback. After that, the driver uses ax88179_in_pm() to determine whether the driver is in the suspend/hibernate path now. Therefore, using a per-driver flag makes sense.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I found
https://elixir.bootlin.com/linux/v7.2-rc6/source/drivers/base/power/
main.c#L
2343
in the Linux source code.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

pm_restrict_gfp_mask() is called before dpm_suspend(state). It removes __GFP_IO and __GFP_FS from the global gfp_t gfp_allowed_mask. Therefore, I think the driver can use GFP_KERNEL safely. Is this correct? Could you please provide any advice?

For the same reason, hibernate_quiet_exec() calls dpm_suspend() directly, so pm_restrict_gfp_mask() is never called. The driver should use GFP_NOIO in this case.

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Andrew

Best,
Chih-Kai

Hi Andrew,

Quoted 8 lines
Quoted 2 lines

Just a gentle ping on this patch.
Based on my previous observation, should the driver use GFP_NOIO

explicitly?

Please try to answer the question yourself. There are lots of drivers out there. What do they do? What does the suspend/resume documentation say?

Andrew

Best Regards,
Chih-Kai