Skip to contentSkip to conversation

atl1 causes lockup/reboot on resume from STR

BUG Applied Luca Tettamanti 5 participants 17 messages Sep 11, 2010 – Sep 22, 2010 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,
I'm having troubles with atl1 driver (notebook is an Asus F3Sa): during the resume from S3 the machine either locks up or reboots. However I was also able to get a (partial) stack trace involving atl1_resume (the rest of the trace scrolled off screen) and a message about a recursive fault, followed by a hard lock up.
If I rmmod atl1 before suspending, the machine comes back fine; I don't known whether this is a regression or not, until recently the radeon driver didn't survive S3 so I didn't even try suspending to RAM... I'm currently running -git current (2.6.36-rc3-00396-gbe6200a), but I've gone back till 2.6.32 without any succcess.
I've no idea on how to debug this, the machine does not have a serial port (and netconsole obviously does not help)... help :)

Luca

#1 lore raw

Luca Tettamanti wrote, On 09/11/2010 03:12 PM:

Quoted 5 lines

Hello,
I'm having troubles with atl1 driver (notebook is an Asus F3Sa): during the resume from S3 the machine either locks up or reboots. However I was also able to get a (partial) stack trace involving atl1_resume (the rest of the trace scrolled off screen) and a message

Quoted 8 lines

about a recursive fault, followed by a hard lock up.
If I rmmod atl1 before suspending, the machine comes back fine; I don't known whether this is a regression or not, until recently the radeon driver didn't survive S3 so I didn't even try suspending to RAM... I'm currently running -git current (2.6.36-rc3-00396-gbe6200a), but I've gone back till 2.6.32 without any succcess.
I've no idea on how to debug this, the machine does not have a serial port (and netconsole obviously does not help)... help :)

Here is an example of kernel photography with an interesting hint by Andrew Morton:
https://bugzilla.kernel.org/show_bug.cgi?id=16626

Jarek P.

#2 lore raw

On Sat, Sep 11, 2010 at 10:47 PM, Jarek Poplawski <[email protected]> wrote:
[...]

Quoted 6 lines
Quoted 2 lines

I've no idea on how to debug this, the machine does not have a serial port (and netconsole obviously does not help)... help :)

Here is an example of kernel photography with an interesting hint by Andrew Morton:
https://bugzilla.kernel.org/show_bug.cgi?id=16626

Thanks for the tip, however the stack trace is extremely elusive: the machine keeps resetting.
Anyway I've managed to track it down the old way (sticking printk everywhere and commenting out random pieces of code :P)

Luca

#3 lore raw
In reply to Luca Tettamanti

Hello,
I've managed to track down the resume bug in the driver, I'm leaving the full quote for reference:

Quoted 5 lines · Luca Tettamanti

Hello,
I'm having troubles with atl1 driver (notebook is an Asus F3Sa): during the resume from S3 the machine either locks up or reboots. However I was also able to get a (partial) stack trace involving atl1_resume (the rest of the trace scrolled off screen) and a message

Quoted 8 lines

about a recursive fault, followed by a hard lock up.
If I rmmod atl1 before suspending, the machine comes back fine; I don't known whether this is a regression or not, until recently the radeon driver didn't survive S3 so I didn't even try suspending to RAM... I'm currently running -git current (2.6.36-rc3-00396-gbe6200a), but I've gone back till 2.6.32 without any succcess.
I've no idea on how to debug this, the machine does not have a serial port (and netconsole obviously does not help)... help :)

The bug is in atl1_resume, when it clears the interrupt status:
adapter->cmb.cmb->int_stats = 0

adapter->cmb.cmb is initialized (atl1_setup_ring_resources) only when the interface is brought up so if the interface is not in use (which is very common for me, since I'm usually on wifi) on resume the kernel will try to dereference a NULL pointer.
Worse still, the memory pointed by adapter->cmb.cmb is freed by atl1_free_ring_resources when the interface is brought down, leaving a stale pointer which is then used in the resume path.
Btw, the same applies to adapter->smb.smb, though it's not used on resume.

The bug should be solved by moving the assignment under "if (netif_running(netdev))" (assuming netif_running does what I think it does :P); it also probably a good idea to reset adapter->cmb.cmb and adapter->smb.smb when the memory is freed anyway.
My notebook has survived a few suspend&resume cycles with and without wired networking, so far so good :)
If you agree on the proposed fixes I'll send the patches.

Luca

#4 lore raw

Ok, I'll be AFK till next monday so here's the patches. I think the first one is suitable for -stable.

Luca

#5 lore raw

PATCH 1/2 atl1: fix resume

In reply to Luca Tettamanti

adapter->cmb.cmb is initialized when the device is opened and freed when it's closed. Accessing it unconditionally during resume results either in a crash (NULL pointer dereference, when the interface has not been opened yet) or data corruption (when the interface has been used and brought down adapter->cmb.cmb points to a deallocated memory area).

---
 drivers/net/atlx/atl1.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)
drivers/net/atlx/atl1.c +3 -2
index 63b9ba0..bbd6e30 100644
@@ -2847,10 +2847,11 @@ static int atl1_resume(struct pci_dev *pdev)
 	pci_enable_wake(pdev, PCI_D3cold, 0);
 
 	atl1_reset_hw(&adapter->hw);
-	adapter->cmb.cmb->int_stats = 0;
 
-	if (netif_running(netdev))
+	if (netif_running(netdev)) {
+		adapter->cmb.cmb->int_stats = 0;
 		atl1_up(adapter);
+	}
 	netif_device_attach(netdev);
 
 	return 0;
Signature · 1 line
1.7.1
Acked-by on v1 1/2
#6 lore raw

Re: [PATCH 1/2] atl1: fix resume

Quoted 30 lines · Luca Tettamanti

adapter->cmb.cmb is initialized when the device is opened and freed when it's closed. Accessing it unconditionally during resume results either in a crash (NULL pointer dereference, when the interface has not been opened yet) or data corruption (when the interface has been used and brought down adapter->cmb.cmb points to a deallocated memory area).

---
 drivers/net/atlx/atl1.c |    5 +++--

 1 files changed, 3 insertions(+), 2 deletions(-)
drivers/net/atlx/atl1.c +3 -3
index 63b9ba0..bbd6e30 100644
@@ -2847,10 +2847,11 @@ static int atl1_resume(struct pci_dev *pdev)
        pci_enable_wake(pdev, PCI_D3cold, 0);
 
        atl1_reset_hw(&adapter->hw);
-       adapter->cmb.cmb->int_stats = 0;
 
-       if (netif_running(netdev))
+       if (netif_running(netdev)) {
+               adapter->cmb.cmb->int_stats = 0;
                atl1_up(adapter);
+       }
        netif_device_attach(netdev);
 
        return 0;
--
Quoted 3 lines

1.7.1

Thanks. I confirmed that the other atlx drivers (which I have been reminded I need to finish merging into a single driver) do not have this bug.

#7 lore raw

Re: [stable] [PATCH 1/2] atl1: fix resume

In reply to Luca Tettamanti
Quoted 8 lines · Luca Tettamanti

adapter->cmb.cmb is initialized when the device is opened and freed when it's closed. Accessing it unconditionally during resume results either in a crash (NULL pointer dereference, when the interface has not been opened yet) or data corruption (when the interface has been used and brought down adapter->cmb.cmb points to a deallocated memory area).

---

You do realize you need to sign-off on a patch you create, right?

thanks,

greg k-h

maintainer
#8 lore raw

Re: [stable] [PATCH 1/2] atl1: fix resume

From: Greg KH <[email protected]>
Date: Thu, 16 Sep 2010 14:13:18 -0700

Quoted 11 lines · Luca Tettamanti
Quoted 8 lines

adapter->cmb.cmb is initialized when the device is opened and freed when it's closed. Accessing it unconditionally during resume results either in a crash (NULL pointer dereference, when the interface has not been opened yet) or data corruption (when the interface has been used and brought down adapter->cmb.cmb points to a deallocated memory area).

---

You do realize you need to sign-off on a patch you create, right?

Right.

Luca please formally resubmit these patches with a proper signoff and all of the accumulated ACKs so far.

#9 lore raw

PATCH 2/2 atl1: zero out CMB and SBM in atl1_free_ring_resources

In reply to Luca Tettamanti

They are allocated in atl1_setup_ring_resources, zero out the pointers in atl1_free_ring_resources (like the other resources).

---
 drivers/net/atlx/atl1.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)
drivers/net/atlx/atl1.c +6 -0
index bbd6e30..c73be28 100644
@@ -1251,6 +1251,12 @@ static void atl1_free_ring_resources(struct atl1_adapter *adapter)
 
 	rrd_ring->desc = NULL;
 	rrd_ring->dma = 0;
+
+	adapter->cmb.dma = 0;
+	adapter->cmb.cmb = NULL;
+
+	adapter->smb.dma = 0;
+	adapter->smb.smb = NULL;
 }
 
 static void atl1_setup_mac_ctrl(struct atl1_adapter *adapter)
Signature · 1 line
1.7.1
Acked-by on v1 2/2
#10 lore raw

Re: [PATCH 2/2] atl1: zero out CMB and SBM in atl1_free_ring_resources

Quoted 24 lines · Luca Tettamanti

They are allocated in atl1_setup_ring_resources, zero out the pointers in atl1_free_ring_resources (like the other resources).
---
 drivers/net/atlx/atl1.c |    6 ++++++

 1 files changed, 6 insertions(+), 0 deletions(-)
drivers/net/atlx/atl1.c +6 -1
index bbd6e30..c73be28 100644
@@ -1251,6 +1251,12 @@ static void atl1_free_ring_resources(struct atl1_adapter *adapter)
 
        rrd_ring->desc = NULL;
        rrd_ring->dma = 0;
+
+       adapter->cmb.dma = 0;
+       adapter->cmb.cmb = NULL;
+
+       adapter->smb.dma = 0;
+       adapter->smb.smb = NULL;
  }
 
  static void atl1_setup_mac_ctrl(struct atl1_adapter *adapter)
--
Quoted 3 lines

1.7.1

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw

Re: [PATCH 1/2] atl1: fix resume

Quoted 9 lines · Luca Tettamanti

adapter->cmb.cmb is initialized when the device is opened and freed when it's closed. Accessing it unconditionally during resume results either in a crash (NULL pointer dereference, when the interface has not been opened yet) or data corruption (when the interface has been used and brought down adapter->cmb.cmb points to a deallocated memory area).

Ping?

Luca

maintainer
#13 lore raw

Re: [PATCH 1/2] atl1: fix resume

From: Luca Tettamanti <[email protected]>
Date: Wed, 22 Sep 2010 21:34:25 +0200

Quoted 7 lines · Luca Tettamanti
Quoted 6 lines

adapter->cmb.cmb is initialized when the device is opened and freed when it's closed. Accessing it unconditionally during resume results either in a crash (NULL pointer dereference, when the interface has not been opened yet) or data corruption (when the interface has been used and brought down adapter->cmb.cmb points to a deallocated memory area).

Quoted 5 lines
Quoted 3 lines

Ping?

For some reason these resubmissions make it to netdev nor patchwork, could you please try sending these two patches again?

Thanks!

#14 lore raw

PATCH 1/2 atl1: fix resume

adapter->cmb.cmb is initialized when the device is opened and freed when it's closed. Accessing it unconditionally during resume results either in a crash (NULL pointer dereference, when the interface has not been opened yet) or data corruption (when the interface has been used and brought down adapter->cmb.cmb points to a deallocated memory area).

---
This time with mutt :)

 drivers/net/atlx/atl1.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)
drivers/net/atlx/atl1.c +3 -2
index 63b9ba0..bbd6e30 100644
@@ -2847,10 +2847,11 @@ static int atl1_resume(struct pci_dev *pdev)
 	pci_enable_wake(pdev, PCI_D3cold, 0);
 
 	atl1_reset_hw(&adapter->hw);
-	adapter->cmb.cmb->int_stats = 0;
 
-	if (netif_running(netdev))
+	if (netif_running(netdev)) {
+		adapter->cmb.cmb->int_stats = 0;
 		atl1_up(adapter);
+	}
 	netif_device_attach(netdev);
 
 	return 0;
Signature · 1 line
1.7.1
maintainer Applied
#15 lore raw

Re: [PATCH 1/2] atl1: fix resume

From: Luca Tettamanti <[email protected]>
Date: Wed, 22 Sep 2010 22:41:58 +0200

Quoted 9 lines

adapter->cmb.cmb is initialized when the device is opened and freed when it's closed. Accessing it unconditionally during resume results either in a crash (NULL pointer dereference, when the interface has not been opened yet) or data corruption (when the interface has been used and brought down adapter->cmb.cmb points to a deallocated memory area).

Applied.

#16 lore raw

PATCH 2/2 atl1: zero out CMB and SBM in atl1_free_ring_resources

In reply to David Miller

They are allocated in atl1_setup_ring_resources, zero out the pointers in atl1_free_ring_resources (like the other resources).

---
 drivers/net/atlx/atl1.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)
drivers/net/atlx/atl1.c +6 -0
index bbd6e30..c73be28 100644
@@ -1251,6 +1251,12 @@ static void atl1_free_ring_resources(struct atl1_adapter *adapter)
 
 	rrd_ring->desc = NULL;
 	rrd_ring->dma = 0;
+
+	adapter->cmb.dma = 0;
+	adapter->cmb.cmb = NULL;
+
+	adapter->smb.dma = 0;
+	adapter->smb.smb = NULL;
 }
 
 static void atl1_setup_mac_ctrl(struct atl1_adapter *adapter)
Signature · 1 line
1.7.1