Skip to contentSkip to conversation

ibmvnic: continue fatal error reset after passive init

PATCH Superseded Lijun Pan 3 participants 6 messages Dec 19, 2020 – Dec 23, 2020

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Lijun Pan

author
#0 lore raw

PATCH 1/1 ibmvnic: continue fatal error reset after passive init

Commit f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") says "If the passive
CRQ initialization occurs before the FATAL reset task is processed, the FATAL error reset task would try to access a CRQ message queue that was freed, causing an oops. The problem may be most likely to occur during DLPAR add vNIC with a non-default MTU, because the DLPAR process will automatically issue a change MTU request.
Fix this by not processing fatal error reset if CRQ is passively initialized after client-driven CRQ initialization fails."

Even with this commit, we still see similar kernel crashes. In order to completely solve this problem, we'd better continue the fatal error reset, capture the kernel crash, and try to fix it from that end.

Fixes f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init")
---
 drivers/net/ethernet/ibm/ibmvnic.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/net/ethernet/ibm/ibmvnic.c +1 -2
index b370c88a43f1..237a36040689 100644
@@ -2342,8 +2342,7 @@ static void __ibmvnic_reset(struct work_struct *work)
 				set_current_state(TASK_UNINTERRUPTIBLE);
 				schedule_timeout(60 * HZ);
 			}
-		} else if (!(rwi->reset_reason == VNIC_RESET_FATAL &&
-				adapter->from_passive_init)) {
+		} else {
 			rc = do_reset(adapter, rwi, reset_state);
 		}
 		kfree(rwi);
Signature · 1 line
2.23.0
maintainer
#1 lore raw
Quoted 10 lines · On Sat, 19 Dec 2020 15:40:34 -0600 Lijun Pan wrote:

Commit f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") says "If the passive
CRQ initialization occurs before the FATAL reset task is processed, the FATAL error reset task would try to access a CRQ message queue that was freed, causing an oops. The problem may be most likely to occur during DLPAR add vNIC with a non-default MTU, because the DLPAR process will automatically issue a change MTU request.
Fix this by not processing fatal error reset if CRQ is passively initialized after client-driven CRQ initialization fails."

Quoted 3 lines

Even with this commit, we still see similar kernel crashes. In order to completely solve this problem, we'd better continue the fatal error reset, capture the kernel crash, and try to fix it from that end.

This basically reverts the quoted fix. Does the quoted fix make things worse? Otherwise we should leave the code be until proper fix is found.

#2 lore raw
Quoted 12 lines · Jakub Kicinski
Quoted 10 lines · On Sat, 19 Dec 2020 15:40:34 -0600 Lijun Pan wrote:

Commit f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") says "If the passive
CRQ initialization occurs before the FATAL reset task is processed, the FATAL error reset task would try to access a CRQ message queue that was freed, causing an oops. The problem may be most likely to occur during DLPAR add vNIC with a non-default MTU, because the DLPAR process will automatically issue a change MTU request.
Fix this by not processing fatal error reset if CRQ is passively initialized after client-driven CRQ initialization fails."

Quoted 6 lines
Quoted 3 lines

Even with this commit, we still see similar kernel crashes. In order to completely solve this problem, we'd better continue the fatal error reset, capture the kernel crash, and try to fix it from that end.

This basically reverts the quoted fix. Does the quoted fix make things worse? Otherwise we should leave the code be until proper fix is found.

Yes, I think the quoted commit makes things worse. It skips the specific reset condition, but that does not fix the problem it claims to fix. The effective fix is upstream SHA 0e435befaea4 and a0faaa27c716. So I think reverting it to the original "else" condition is the right thing to do.

maintainer
#3 lore raw
Quoted 16 lines · On Wed, 23 Dec 2020 02:21:09 -0600 Lijun Pan wrote:
Quoted 15 lines · Jakub Kicinski
Quoted 13 lines · On Sat, 19 Dec 2020 15:40:34 -0600 Lijun Pan wrote:

Commit f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") says "If the passive
CRQ initialization occurs before the FATAL reset task is processed, the FATAL error reset task would try to access a CRQ message queue that was freed, causing an oops. The problem may be most likely to occur during DLPAR add vNIC with a non-default MTU, because the DLPAR process will automatically issue a change MTU request.
Fix this by not processing fatal error reset if CRQ is passively initialized after client-driven CRQ initialization fails."

Even with this commit, we still see similar kernel crashes. In order to completely solve this problem, we'd better continue the fatal error reset, capture the kernel crash, and try to fix it from that end.

Quoted 5 lines
Quoted 2 lines

This basically reverts the quoted fix. Does the quoted fix make things worse? Otherwise we should leave the code be until proper fix is found.

Yes, I think the quoted commit makes things worse. It skips the specific reset condition, but that does not fix the problem it claims to fix.

Okay, let's make sure the commit message explains how it makes things worse.

Quoted 2 lines

The effective fix is upstream SHA 0e435befaea4 and a0faaa27c716. So I think reverting it to the original "else" condition is the right thing to do.

Hm. So the problem is fixed? But the commit message says "we still see similar kernel crashes", that's present tense suggesting that crashes are seen on current net/master. Are you saying that's not the case and after 0e435befaea4 and a0faaa27c716 there are no more crashes?

#4 lore raw
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Wed, 23 Dec 2020 02:21:09 -0600 Lijun Pan wrote:
Quoted 15 lines · Jakub Kicinski
Quoted 13 lines · On Sat, 19 Dec 2020 15:40:34 -0600 Lijun Pan wrote:

Commit f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") says "If the passive
CRQ initialization occurs before the FATAL reset task is processed, the FATAL error reset task would try to access a CRQ message queue that was freed, causing an oops. The problem may be most likely to occur during DLPAR add vNIC with a non-default MTU, because the DLPAR process will automatically issue a change MTU request.
Fix this by not processing fatal error reset if CRQ is passively initialized after client-driven CRQ initialization fails."

Even with this commit, we still see similar kernel crashes. In order to completely solve this problem, we'd better continue the fatal error reset, capture the kernel crash, and try to fix it from that end.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

This basically reverts the quoted fix. Does the quoted fix make things worse? Otherwise we should leave the code be until proper fix is found.

Yes, I think the quoted commit makes things worse. It skips the specific reset condition, but that does not fix the problem it claims to fix.

Okay, let's make sure the commit message explains how it makes things worse.

I will reword the commit message.

Quoted 8 lines
Quoted 3 lines

The effective fix is upstream SHA 0e435befaea4 and a0faaa27c716. So I think reverting it to the original "else" condition is the right thing to do.

Hm. So the problem is fixed? But the commit message says "we still see similar kernel crashes", that's present tense suggesting that crashes are seen on current net/master. Are you saying that's not the case and after 0e435befaea4 and a0faaa27c716 there are no more crashes?

This patch was formed before I submitted 0e435befaea4 and a0faaa27c716, so I used the wording "we still see similar kernel crashes". I will modify the commit message before I submit v2 of this patch.
After 0e435befaea4 and a0faaa27c716, I don't see any crashes as described in this quoted commit even without this quoted commit.
That's why I am sure this quoted commit does not fix the described problem and I want to revert it.

maintainer
#5 lore raw
Quoted 38 lines · On Wed, 23 Dec 2020 14:10:32 -0600 Lijun Pan wrote:
Quoted 18 lines · Jakub Kicinski
Quoted 16 lines · On Wed, 23 Dec 2020 02:21:09 -0600 Lijun Pan wrote:
Quoted 15 lines · Jakub Kicinski
Quoted 13 lines · On Sat, 19 Dec 2020 15:40:34 -0600 Lijun Pan wrote:

Commit f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") says "If the passive
CRQ initialization occurs before the FATAL reset task is processed, the FATAL error reset task would try to access a CRQ message queue that was freed, causing an oops. The problem may be most likely to occur during DLPAR add vNIC with a non-default MTU, because the DLPAR process will automatically issue a change MTU request.
Fix this by not processing fatal error reset if CRQ is passively initialized after client-driven CRQ initialization fails."

Even with this commit, we still see similar kernel crashes. In order to completely solve this problem, we'd better continue the fatal error reset, capture the kernel crash, and try to fix it from that end.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

This basically reverts the quoted fix. Does the quoted fix make things worse? Otherwise we should leave the code be until proper fix is found.

Yes, I think the quoted commit makes things worse. It skips the specific reset condition, but that does not fix the problem it claims to fix.

Okay, let's make sure the commit message explains how it makes things worse.

I will reword the commit message.

Quoted 7 lines
Quoted 2 lines

The effective fix is upstream SHA 0e435befaea4 and a0faaa27c716. So I think reverting it to the original "else" condition is the right thing to do.

Hm. So the problem is fixed? But the commit message says "we still see similar kernel crashes", that's present tense suggesting that crashes are seen on current net/master. Are you saying that's not the case and after 0e435befaea4 and a0faaa27c716 there are no more crashes?

Quoted 7 lines

This patch was formed before I submitted 0e435befaea4 and a0faaa27c716, so I used the wording "we still see similar kernel crashes". I will modify the commit message before I submit v2 of this patch.
After 0e435befaea4 and a0faaa27c716, I don't see any crashes as described in this quoted commit even without this quoted commit.
That's why I am sure this quoted commit does not fix the described problem and I want to revert it.

I see, that explains it!