Skip to contentSkip to conversation

qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()

PATCH Applied Jia He 4 participants 9 messages Jul 15, 2021 – Jul 20, 2021

Subsystems (3)
lore
Share & tools
lore

Jia He

author
#0 lore raw

PATCH 1/1 qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()

Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ QL41000 ethernet controller:

 BUG: scheduling while atomic: kworker/0:4/531/0x00000200
  [qed_probe:488()]hw prepare failed
  kernel BUG at mm/vmalloc.c:2355!
  Internal error: Oops - BUG: 0 [#1] SMP
  CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-Ubuntu
  pstate: 00400009 (nzcv daif +PAN -UAO)
 Call trace:
  vunmap+0x4c/0x50
  iounmap+0x48/0x58
  qed_free_pci+0x60/0x80 [qed]
  qed_probe+0x35c/0x688 [qed]
  __qede_probe+0x88/0x5c8 [qede]
  qede_probe+0x60/0xe0 [qede]
  local_pci_probe+0x48/0xa0
  work_for_cpu_fn+0x24/0x38
  process_one_work+0x1d0/0x468
  worker_thread+0x238/0x4e0
  kthread+0xf0/0x118
  ret_from_fork+0x10/0x18

In this case, qed_hw_prepare() returns error due to hw/fw error, but in theory work queue should be in process context instead of interrupt.

The root cause might be the unpaired spin_{un}lock_bh() in _qed_mcp_cmd_and_union(), which causes botton half is disabled incorrectly.

---
 drivers/net/ethernet/qlogic/qed/qed_mcp.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)
drivers/net/ethernet/qlogic/qed/qed_mcp.c +17 -6
index 4387292c37e2..79d879a5d663 100644
@@ -474,14 +474,18 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
 
 		spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
 
-		if (!qed_mcp_has_pending_cmd(p_hwfn))
+		if (!qed_mcp_has_pending_cmd(p_hwfn)) {
+			spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 			break;
+		}
 
 		rc = qed_mcp_update_pending_cmd(p_hwfn, p_ptt);
-		if (!rc)
+		if (!rc) {
+			spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 			break;
-		else if (rc != -EAGAIN)
+		} else if (rc != -EAGAIN) {
 			goto err;
+		}
 
 		spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 
@@ -498,6 +502,8 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
 		return -EAGAIN;
 	}
 
+	spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
+
 	/* Send the mailbox command */
 	qed_mcp_reread_offsets(p_hwfn, p_ptt);
 	seq_num = ++p_hwfn->mcp_info->drv_mb_seq;
@@ -524,14 +530,18 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
 
 		spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
 
-		if (p_cmd_elem->b_is_completed)
+		if (p_cmd_elem->b_is_completed) {
+			spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 			break;
+		}
 
 		rc = qed_mcp_update_pending_cmd(p_hwfn, p_ptt);
-		if (!rc)
+		if (!rc) {
+			spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 			break;
-		else if (rc != -EAGAIN)
+		} else if (rc != -EAGAIN) {
 			goto err;
+		}
 
 		spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 	} while (++cnt < max_retries);
@@ -554,6 +564,7 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
 		return -EAGAIN;
 	}
 
+	spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
 	qed_mcp_cmd_del_elem(p_hwfn, p_cmd_elem);
 	spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
 
Signature · 1 line
2.17.1
#1 lore raw
This patch was applied to netdev/net.git (refs/heads/master):

Hello:

This patch was applied to netdev/net.git (refs/heads/master):

Quoted 22 lines · On Thu, 15 Jul 2021 16:08:21 +0800 you wrote:

Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ QL41000 ethernet controller:

 BUG: scheduling while atomic: kworker/0:4/531/0x00000200
  [qed_probe:488()]hw prepare failed
  kernel BUG at mm/vmalloc.c:2355!
  Internal error: Oops - BUG: 0 [#1] SMP
  CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-Ubuntu
  pstate: 00400009 (nzcv daif +PAN -UAO)
 Call trace:
  vunmap+0x4c/0x50
  iounmap+0x48/0x58
  qed_free_pci+0x60/0x80 [qed]
  qed_probe+0x35c/0x688 [qed]
  __qede_probe+0x88/0x5c8 [qede]
  qede_probe+0x60/0xe0 [qede]
  local_pci_probe+0x48/0xa0
  work_for_cpu_fn+0x24/0x38
  process_one_work+0x1d0/0x468
  worker_thread+0x238/0x4e0
  kthread+0xf0/0x118
  ret_from_fork+0x10/0x18
Quoted 1 line

[...]

Here is the summary with links:
- qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()

    https://git.kernel.org/netdev/net/c/6206b7981a36

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

author
#2 lore raw
Quoted 41 lines

-----Original Message-----
From: [email protected] <patchwork- [email protected]>
Sent: Friday, July 16, 2021 3:50 AM
To: Justin He <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; nd <[email protected]>
Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()

Hello:

This patch was applied to netdev/net.git (refs/heads/master):

Quoted 7 lines · On Thu, 15 Jul 2021 16:08:21 +0800 you wrote:

Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ QL41000 ethernet controller:

 BUG: scheduling while atomic: kworker/0:4/531/0x00000200
  [qed_probe:488()]hw prepare failed
  kernel BUG at mm/vmalloc.c:2355!
  Internal error: Oops - BUG: 0 [#1] SMP
  CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-

Ubuntu

Quoted 15 lines

pstate: 00400009 (nzcv daif +PAN -UAO)

 Call trace:
  vunmap+0x4c/0x50
  iounmap+0x48/0x58
  qed_free_pci+0x60/0x80 [qed]
  qed_probe+0x35c/0x688 [qed]
  __qede_probe+0x88/0x5c8 [qede]
  qede_probe+0x60/0xe0 [qede]
  local_pci_probe+0x48/0xa0
  work_for_cpu_fn+0x24/0x38
  process_one_work+0x1d0/0x468
  worker_thread+0x238/0x4e0
  kthread+0xf0/0x118
  ret_from_fork+0x10/0x18
Quoted 7 lines
Quoted 1 line

[...]

Here is the summary with links:
- qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()

    https://git.kernel.org/netdev/net/c/6206b7981a36

You are awesome, thank you!

Thanks.
If possible, please also Cc: [email protected] because the bug seemed to be there for a long time.

--
Cheers,
Justin (Jia He)

#3 lore raw
In reply to Jia He

Hi Jia,

Quoted 26 lines · Jia He

Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ QL41000 ethernet controller:

 BUG: scheduling while atomic: kworker/0:4/531/0x00000200
  [qed_probe:488()]hw prepare failed
  kernel BUG at mm/vmalloc.c:2355!
  Internal error: Oops - BUG: 0 [#1] SMP
  CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-Ubuntu
  pstate: 00400009 (nzcv daif +PAN -UAO)
 Call trace:
  vunmap+0x4c/0x50
  iounmap+0x48/0x58
  qed_free_pci+0x60/0x80 [qed]
  qed_probe+0x35c/0x688 [qed]
  __qede_probe+0x88/0x5c8 [qede]
  qede_probe+0x60/0xe0 [qede]
  local_pci_probe+0x48/0xa0
  work_for_cpu_fn+0x24/0x38
  process_one_work+0x1d0/0x468
  worker_thread+0x238/0x4e0
  kthread+0xf0/0x118
  ret_from_fork+0x10/0x18

In this case, qed_hw_prepare() returns error due to hw/fw error, but in theory work queue should be in process context instead of interrupt.

Quoted 6 lines

The root cause might be the unpaired spin_{un}lock_bh() in _qed_mcp_cmd_and_union(), which causes botton half is disabled incorrectly.

---

This patch is adding additional spin_{un}lock_bh().
Can you please enlighten about the exact flow causing this unpaired spin_{un}lock_bh.

Also,
as per description, looks like you are not sure actual the root-cause. does this patch really solved the problem?

--pk

author
#4 lore raw

Hi Prabhakar

Quoted 46 lines

-----Original Message-----
From: Prabhakar Kushwaha <[email protected]>
Sent: Monday, July 19, 2021 6:36 PM
To: Justin He <[email protected]>
Cc: Ariel Elior <[email protected]>; [email protected]; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; [email protected]; Linux Kernel Mailing List <linux- [email protected]>; nd <[email protected]>; Shai Malin <[email protected]>; Shai Malin <[email protected]>; Prabhakar Kushwaha <[email protected]>
Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()

Hi Jia,

Quoted 8 lines · Jia He

Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ QL41000 ethernet controller:

 BUG: scheduling while atomic: kworker/0:4/531/0x00000200
  [qed_probe:488()]hw prepare failed
  kernel BUG at mm/vmalloc.c:2355!
  Internal error: Oops - BUG: 0 [#1] SMP
  CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-

Ubuntu

Quoted 15 lines

pstate: 00400009 (nzcv daif +PAN -UAO)

 Call trace:
  vunmap+0x4c/0x50
  iounmap+0x48/0x58
  qed_free_pci+0x60/0x80 [qed]
  qed_probe+0x35c/0x688 [qed]
  __qede_probe+0x88/0x5c8 [qede]
  qede_probe+0x60/0xe0 [qede]
  local_pci_probe+0x48/0xa0
  work_for_cpu_fn+0x24/0x38
  process_one_work+0x1d0/0x468
  worker_thread+0x238/0x4e0
  kthread+0xf0/0x118
  ret_from_fork+0x10/0x18
Quoted 5 lines

In this case, qed_hw_prepare() returns error due to hw/fw error, but in theory work queue should be in process context instead of interrupt.

The root cause might be the unpaired spin_{un}lock_bh() in _qed_mcp_cmd_and_union(), which causes botton half is disabled

incorrectly.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

---

This patch is adding additional spin_{un}lock_bh().
Can you please enlighten about the exact flow causing this unpaired spin_{un}lock_bh.

For instance:
_qed_mcp_cmd_and_union()
In while loop
spin_lock_bh()
qed_mcp_has_pending_cmd() (assume false), will break the loop
if (cnt >= max_retries) {
...
return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
}

Quoted 3 lines

Also,
as per description, looks like you are not sure actual the root-cause. does this patch really solved the problem?

I don't have that ThunderX2 to verify the patch.
But I searched all the spin_lock/unlock_bh and spin_lock_irqsave/irqrestore under driver/.../qlogic, this is the only problematic point I could figure out. And this might be possible code path of qed_probe().

--
Cheers,
Justin (Jia He)

#5 lore raw

Hi Justin,

Quoted 57 lines · Justin He

Hi Prabhakar

Quoted 46 lines

-----Original Message-----
From: Prabhakar Kushwaha <[email protected]>
Sent: Monday, July 19, 2021 6:36 PM
To: Justin He <[email protected]>
Cc: Ariel Elior <[email protected]>; [email protected]; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; [email protected]; Linux Kernel Mailing List <linux- [email protected]>; nd <[email protected]>; Shai Malin <[email protected]>; Shai Malin <[email protected]>; Prabhakar Kushwaha <[email protected]>
Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()

Hi Jia,

Quoted 8 lines · Jia He

Liajian reported a bug_on hit on a ThunderX2 arm64 server with FastLinQ QL41000 ethernet controller:

 BUG: scheduling while atomic: kworker/0:4/531/0x00000200
  [qed_probe:488()]hw prepare failed
  kernel BUG at mm/vmalloc.c:2355!
  Internal error: Oops - BUG: 0 [#1] SMP
  CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic #86-

Ubuntu

Quoted 15 lines

pstate: 00400009 (nzcv daif +PAN -UAO)

 Call trace:
  vunmap+0x4c/0x50
  iounmap+0x48/0x58
  qed_free_pci+0x60/0x80 [qed]
  qed_probe+0x35c/0x688 [qed]
  __qede_probe+0x88/0x5c8 [qede]
  qede_probe+0x60/0xe0 [qede]
  local_pci_probe+0x48/0xa0
  work_for_cpu_fn+0x24/0x38
  process_one_work+0x1d0/0x468
  worker_thread+0x238/0x4e0
  kthread+0xf0/0x118
  ret_from_fork+0x10/0x18
Quoted 5 lines

In this case, qed_hw_prepare() returns error due to hw/fw error, but in theory work queue should be in process context instead of interrupt.

The root cause might be the unpaired spin_{un}lock_bh() in _qed_mcp_cmd_and_union(), which causes botton half is disabled

incorrectly.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

---

This patch is adding additional spin_{un}lock_bh().
Can you please enlighten about the exact flow causing this unpaired spin_{un}lock_bh.

Quoted 5 lines

For instance:
_qed_mcp_cmd_and_union()
In while loop
spin_lock_bh()
qed_mcp_has_pending_cmd() (assume false), will break the loop

I agree till here.

Quoted 5 lines

if (cnt >= max_retries) {
...
return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
}

Because of break, cnt has not been increased.
- cnt is still less than max_retries.
- if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
Hence pairing completed.

I am not seeing any issue here.

Quoted 9 lines
Quoted 3 lines

Also,
as per description, looks like you are not sure actual the root-cause. does this patch really solved the problem?

I don't have that ThunderX2 to verify the patch.
But I searched all the spin_lock/unlock_bh and spin_lock_irqsave/irqrestore under driver/.../qlogic, this is the only problematic point I could figure out. And this might be possible code path of qed_probe().

Without testing and proper root-cause, it is tough to accept the suggested fix.

--pk

author
#6 lore raw
Quoted 93 lines

-----Original Message-----
From: Prabhakar Kushwaha <[email protected]>
Sent: Monday, July 19, 2021 10:51 PM
To: Justin He <[email protected]>
Cc: Ariel Elior <[email protected]>; [email protected]; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; [email protected]; Linux Kernel Mailing List <linux- [email protected]>; nd <[email protected]>; Shai Malin <[email protected]>; Shai Malin <[email protected]>; Prabhakar Kushwaha <[email protected]>
Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()

Hi Justin,

Quoted 11 lines · Justin He

Hi Prabhakar

Quoted 8 lines

-----Original Message-----
From: Prabhakar Kushwaha <[email protected]>
Sent: Monday, July 19, 2021 6:36 PM
To: Justin He <[email protected]>
Cc: Ariel Elior <[email protected]>; [email protected]; David S. Miller <[email protected]>; Jakub Kicinski <[email protected]>; [email protected]; Linux Kernel Mailing List <linux- [email protected]>; nd <[email protected]>; Shai Malin

<[email protected]>;

Quoted 1 line
Quoted 1 line

Shai Malin <[email protected]>; Prabhakar Kushwaha

<[email protected]>

Quoted 8 lines
Quoted 8 lines

Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()

Hi Jia,

Quoted 2 lines · Jia He

Liajian reported a bug_on hit on a ThunderX2 arm64 server with

FastLinQ

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

QL41000 ethernet controller:

 BUG: scheduling while atomic: kworker/0:4/531/0x00000200
  [qed_probe:488()]hw prepare failed
  kernel BUG at mm/vmalloc.c:2355!
  Internal error: Oops - BUG: 0 [#1] SMP
  CPU: 0 PID: 531 Comm: kworker/0:4 Tainted: G W 5.4.0-77-generic

#86-

Quoted 16 lines
Quoted 16 lines

Ubuntu

Quoted 15 lines

pstate: 00400009 (nzcv daif +PAN -UAO)

 Call trace:
  vunmap+0x4c/0x50
  iounmap+0x48/0x58
  qed_free_pci+0x60/0x80 [qed]
  qed_probe+0x35c/0x688 [qed]
  __qede_probe+0x88/0x5c8 [qede]
  qede_probe+0x60/0xe0 [qede]
  local_pci_probe+0x48/0xa0
  work_for_cpu_fn+0x24/0x38
  process_one_work+0x1d0/0x468
  worker_thread+0x238/0x4e0
  kthread+0xf0/0x118
  ret_from_fork+0x10/0x18
Quoted 1 line
Quoted 1 line
Quoted 1 line

In this case, qed_hw_prepare() returns error due to hw/fw error, but

in

Quoted 14 lines
Quoted 6 lines
Quoted 4 lines

theory work queue should be in process context instead of interrupt.

The root cause might be the unpaired spin_{un}lock_bh() in _qed_mcp_cmd_and_union(), which causes botton half is disabled

incorrectly.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

---

This patch is adding additional spin_{un}lock_bh().
Can you please enlighten about the exact flow causing this unpaired spin_{un}lock_bh.

Quoted 5 lines

For instance:
_qed_mcp_cmd_and_union()
In while loop
spin_lock_bh()
qed_mcp_has_pending_cmd() (assume false), will break the loop

I agree till here.

Quoted 5 lines

if (cnt >= max_retries) {
...
return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
}

Quoted 4 lines

Because of break, cnt has not been increased.
- cnt is still less than max_retries.
- if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
Hence pairing completed.

Sorry, indeed. Let me check other possibilities.
@David S. Miller Sorry for the inconvenience, could you please revert it in netdev tree?

Apologies again.

--
Cheers,
Justin (Jia He)

maintainer
#7 lore raw
Quoted 14 lines · Justin He
Quoted 14 lines
Quoted 5 lines

For instance:
_qed_mcp_cmd_and_union()
In while loop
spin_lock_bh()
qed_mcp_has_pending_cmd() (assume false), will break the loop

I agree till here.

Quoted 5 lines

if (cnt >= max_retries) {
...
return -EAGAIN; <-- here returns -EAGAIN without invoking bh unlock
}

Quoted 8 lines
Quoted 4 lines

Because of break, cnt has not been increased.
- cnt is still less than max_retries.
- if (cnt >= max_retries) will not be *true*, leading to spin_unlock_bh().
Hence pairing completed.

Sorry, indeed. Let me check other possibilities.
@David S. Miller Sorry for the inconvenience, could you please revert it in netdev tree?

Please submit a revert patch with the conclusions from the discussion included in the commit message.

author
#8 lore raw
Quoted 34 lines

-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: Tuesday, July 20, 2021 5:06 PM
To: Justin He <[email protected]>
Cc: Prabhakar Kushwaha <[email protected]>; David S. Miller <[email protected]>; Ariel Elior <[email protected]>; GR-everest-linux- [email protected]; [email protected]; Linux Kernel Mailing List <linux- [email protected]>; nd <[email protected]>; Shai Malin <[email protected]>; Shai Malin <[email protected]>; Prabhakar Kushwaha <[email protected]>
Subject: Re: [PATCH] qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()

Quoted 11 lines · Justin He
Quoted 11 lines
Quoted 5 lines

For instance:
_qed_mcp_cmd_and_union()
In while loop
spin_lock_bh()
qed_mcp_has_pending_cmd() (assume false), will break the loop

I agree till here.

Quoted 3 lines

if (cnt >= max_retries) {
...
return -EAGAIN; <-- here returns -EAGAIN without invoking bh

unlock

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
  }

Because of break, cnt has not been increased.
- cnt is still less than max_retries.
- if (cnt >= max_retries) will not be *true*, leading to

spin_unlock_bh().

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Hence pairing completed.

Sorry, indeed. Let me check other possibilities.
@David S. Miller Sorry for the inconvenience, could you please revert it in netdev tree?

Please submit a revert patch with the conclusions from the discussion included in the commit message.

Okay,will do that
Thanks for the reminder

--
Cheers,
Justin (Jia He)