Skip to contentSkip to conversation

net: atlantic: always deep reset on pm op, fixing null deref regression

PATCH Superseded Manuel Ullmann 5 participants 5 messages May 4, 2022 – May 5, 2022

Subsystems (4)
Networking Network drivers Regressions Aquantia Ethernet driver (atlantic)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Manuel Ullmann

author
#0 lore raw

PATCH 1/1 net: atlantic: always deep reset on pm op, fixing null deref regression

From a3eccd32c618fe4b4f5c537cd83ba5611149623e Mon Sep 17 00:00:00 2001
Date: Wed, 4 May 2022 21:30:44 +0200

The impact of this regression is the same for resume that I saw on
thaw: the kernel hangs and nothing except SysRq rebooting can be done.

The null deref occurs at the same position as on thaw.

BUG: kernel NULL pointer dereference
RIP: aq_ring_rx_fill+0xcf/0x210 [atlantic]

Fixes regression in cbe6c3a8f8f4 ("net: atlantic: invert deep par in pm functions, preventing null derefs"), where I disabled deep pm resets in suspend and resume, trying to make sense of the atl_resume_common deep parameter in the first place.

It turns out, that atlantic always has to deep reset on pm operations and the parameter is useless. Even though I expected that and tested resume, I screwed up by kexec-rebooting into an unpatched kernel, thus missing the breakage.

This fixup obsoletes the deep parameter of atl_resume_common, but I leave the cleanup for the maintainers to post to mainline.

PS: I'm very sorry for this regression.

Fixes cbe6c3a8f8f4315b96e46e1a1c70393c06d95a4c
Link https://lore.kernel.org/regressions/9-Ehc_xXSwdXcvZqKD5aSqsqeNj5Izco4MYEwnx5cySXVEc9-x_WC4C3kAoCqNTi-H38frroUK17iobNVnkLtW36V6VWGSQEOHXhmVMm5iQ=@protonmail.com/ open here
Reported-by Jordan Leppert <[email protected]>
Reported-by Holger Hoffstätte <[email protected]>
CC <[email protected]> # 5.17.5
CC <[email protected]> # 5.15.36
CC <[email protected]> # 5.10.113
Signed-off-by Manuel ULlmann <[email protected]>
---
 drivers/net/ethernet/aquantia/atlantic/aq_pci_func.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/aquantia/atlantic/aq_pci_func.c +2 -2
index 3a529ee8c834..831833911a52 100644
@@ -449,7 +449,7 @@ static int aq_pm_freeze(struct device *dev)
 
 static int aq_pm_suspend_poweroff(struct device *dev)
 {
-	return aq_suspend_common(dev, false);
+	return aq_suspend_common(dev, true);
 }
 
 static int aq_pm_thaw(struct device *dev)

@@ -459,7 +459,7 @@ static int aq_pm_thaw(struct device *dev)
 
 static int aq_pm_resume_restore(struct device *dev)
 {
-	return atl_resume_common(dev, false);
+	return atl_resume_common(dev, true);
 }
 
 static const struct dev_pm_ops aq_pm_ops = {

base-commit: 672c0c5173427e6b3e2a9bbb7be51ceeec78093a

Signature · 1 line
2.35.1
Tested-by on v1 1/1
#1 lore raw
Quoted 60 lines · Manuel Ullmann

From a3eccd32c618fe4b4f5c537cd83ba5611149623e Mon Sep 17 00:00:00 2001
Date: Wed, 4 May 2022 21:30:44 +0200

The impact of this regression is the same for resume that I saw on
thaw: the kernel hangs and nothing except SysRq rebooting can be done.

The null deref occurs at the same position as on thaw.

BUG: kernel NULL pointer dereference
RIP: aq_ring_rx_fill+0xcf/0x210 [atlantic]

Fixes regression in cbe6c3a8f8f4 ("net: atlantic: invert deep par in pm functions, preventing null derefs"), where I disabled deep pm resets in suspend and resume, trying to make sense of the atl_resume_common deep parameter in the first place.

It turns out, that atlantic always has to deep reset on pm operations and the parameter is useless. Even though I expected that and tested resume, I screwed up by kexec-rebooting into an unpatched kernel, thus missing the breakage.

This fixup obsoletes the deep parameter of atl_resume_common, but I leave the cleanup for the maintainers to post to mainline.

PS: I'm very sorry for this regression.

Fixes cbe6c3a8f8f4315b96e46e1a1c70393c06d95a4c
Link https://lore.kernel.org/regressions/9-Ehc_xXSwdXcvZqKD5aSqsqeNj5Izco4MYEwnx5cySXVEc9-x_WC4C3kAoCqNTi-H38frroUK17iobNVnkLtW36V6VWGSQEOHXhmVMm5iQ=@protonmail.com/ open here
Reported-by Jordan Leppert <[email protected]>
Reported-by Holger Hoffstätte <[email protected]>
CC <[email protected]> # 5.17.5
CC <[email protected]> # 5.15.36
CC <[email protected]> # 5.10.113
Signed-off-by Manuel ULlmann <[email protected]>
---
  drivers/net/ethernet/aquantia/atlantic/aq_pci_func.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/aquantia/atlantic/aq_pci_func.c +2 -2
index 3a529ee8c834..831833911a52 100644
@@ -449,7 +449,7 @@ static int aq_pm_freeze(struct device *dev)
 
  static int aq_pm_suspend_poweroff(struct device *dev)
  {
-	return aq_suspend_common(dev, false);
+	return aq_suspend_common(dev, true);
  }
 
  static int aq_pm_thaw(struct device *dev)

@@ -459,7 +459,7 @@ static int aq_pm_thaw(struct device *dev)
 
  static int aq_pm_resume_restore(struct device *dev)
  {
-	return atl_resume_common(dev, false);
+	return atl_resume_common(dev, true);
  }
 
  static const struct dev_pm_ops aq_pm_ops = {
Quoted 2 lines

base-commit: 672c0c5173427e6b3e2a9bbb7be51ceeec78093a

As mentioned in the discusson thread this reliably restores resume for me, so:

Tested-by Holger Hoffstätte <[email protected]>

Thanks!
Holger

#2 lore raw
In reply to Manuel Ullmann
Quoted 18 lines

The impact of this regression is the same for resume that I saw on
thaw: the kernel hangs and nothing except SysRq rebooting can be done.

The null deref occurs at the same position as on thaw.

BUG: kernel NULL pointer dereference
RIP: aq_ring_rx_fill+0xcf/0x210 [atlantic]

Fixes regression in cbe6c3a8f8f4 ("net: atlantic: invert deep par in pm functions, preventing null derefs"), where I disabled deep pm resets in suspend and resume, trying to make sense of the atl_resume_common deep parameter in the first place.

It turns out, that atlantic always has to deep reset on pm operations and the parameter is useless. Even though I expected that and tested resume, I screwed up by kexec-rebooting into an unpatched kernel, thus missing the breakage.

Quoted 4 lines

This fixup obsoletes the deep parameter of atl_resume_common, but I leave the cleanup for the maintainers to post to mainline.

PS: I'm very sorry for this regression.

Hi Manuel,

Unfortunately I've missed to review and comment on previous patch - it was too quickly accepted.

I'm still in doubt on your fixes, even after rereading the original problem. Is it possible for you to test this with all the possible combinations? suspend/resume with device up/down,
hibernate/restore with device up/down?

I'll try to do the same on our side, but we don't have much resources for that now unfortunately..

Quoted 1 line
Fixes cbe6c3a8f8f4315b96e46e1a1c70393c06d95a4c

That tag format is incorrect I think..

Igor

#3 lore raw

With the proposed patch (deep parameter is always true), I've managed to test:
1. Hibernate/restore (with device down/up)
2. Suspend/resume (with device down/up)

I put the device down with the command:
sudo ip link set <connection> down

I hope that's correct, if not please let me know correct command.

Regards,
Jordan

------- Original Message -------

Quoted 42 lines · Igor Russkikh
Quoted 19 lines

The impact of this regression is the same for resume that I saw on
thaw: the kernel hangs and nothing except SysRq rebooting can be done.

The null deref occurs at the same position as on thaw.

BUG: kernel NULL pointer dereference
RIP: aq_ring_rx_fill+0xcf/0x210 [atlantic]

Fixes regression in cbe6c3a8f8f4 ("net: atlantic: invert deep par in pm functions, preventing null derefs"), where I disabled deep pm resets in suspend and resume, trying to make sense of the atl_resume_common deep parameter in the first place.

It turns out, that atlantic always has to deep reset on pm operations and the parameter is useless. Even though I expected that and tested resume, I screwed up by kexec-rebooting into an unpatched kernel, thus missing the breakage.

Quoted 4 lines

This fixup obsoletes the deep parameter of atl_resume_common, but I leave the cleanup for the maintainers to post to mainline.

PS: I'm very sorry for this regression.

Hi Manuel,

Unfortunately I've missed to review and comment on previous patch - it was too quickly accepted.

I'm still in doubt on your fixes, even after rereading the original problem. Is it possible for you to test this with all the possible combinations? suspend/resume with device up/down,
hibernate/restore with device up/down?

I'll try to do the same on our side, but we don't have much resources for that now unfortunately..

Quoted 1 line
Fixes cbe6c3a8f8f4315b96e46e1a1c70393c06d95a4c

That tag format is incorrect I think..

Igor

#4 lore raw
Quoted 25 lines
Quoted 25 lines
Quoted 23 lines

The impact of this regression is the same for resume that I saw on
thaw: the kernel hangs and nothing except SysRq rebooting can be done.

The null deref occurs at the same position as on thaw.

BUG: kernel NULL pointer dereference
RIP: aq_ring_rx_fill+0xcf/0x210 [atlantic]

Fixes regression in cbe6c3a8f8f4 ("net: atlantic: invert deep par in pm functions, preventing null derefs"), where I disabled deep pm resets in suspend and resume, trying to make sense of the atl_resume_common deep parameter in the first place.

It turns out, that atlantic always has to deep reset on pm operations and the parameter is useless. Even though I expected that and tested resume, I screwed up by kexec-rebooting into an unpatched kernel, thus missing the breakage.

This fixup obsoletes the deep parameter of atl_resume_common, but I leave the cleanup for the maintainers to post to mainline.

PS: I'm very sorry for this regression.

Quoted 8 lines
Quoted 8 lines

Hi Manuel,

Unfortunately I've missed to review and comment on previous patch - it was too quickly accepted.

I'm still in doubt on your fixes, even after rereading the original problem. Is it possible for you to test this with all the possible combinations? suspend/resume with device up/down,
hibernate/restore with device up/down?

I confirm that suspend/resume/hibernation/thaw keeps working in all cases. Thaw would work without the original patch, if the device is down before hibernation. I also originally described this behaviour on bugzilla at

https://bugzilla.kernel.org/show_bug.cgi?id=215798

See also Jordan’s confirmation below.

I think, the main reason, why this could break, is, that the deep parameter had no real impact until the breaking commit. So it was practically untested, when the allocation/free functions were split.

Another thing, that I tested, was guarding all null pointer references with null checks, which failed at first, because GCC optimized them out. I think I have the atlantic tree for this (bad) fix attempt floating around. I can try to rebase and create a patch from this and post it to the Github issue, if you are interested.
https://github.com/Aquantia/AQtion/issues/32

Don’t have time for this before the weekend though.

Quoted 6 lines
Quoted 6 lines

I'll try to do the same on our side, but we don't have much resources for that now unfortunately..

Quoted 1 line
Fixes cbe6c3a8f8f4315b96e46e1a1c70393c06d95a4c

That tag format is incorrect I think..

Thanks for pointing that out. Also, are those stable Cc tags correct? Because I figured, that the x in the documentation could be also the branch name and not a placeholder. Should I resend the patch, fixing the tags? Won’t get to it before tomorrow, though.

Quoted 10 lines
Quoted 1 line

Igor

With the proposed patch (deep parameter is always true), I've managed to test:
1. Hibernate/restore (with device down/up)
2. Suspend/resume (with device down/up)

I put the device down with the command:
sudo ip link set <connection> down

I hope that's correct, if not please let me know correct command.

This should be the correct.

Manuel