Skip to contentSkip to conversation

wireless: ath10k: Return early in ath10k_qmi_event_server_exit() to avoid hard crash on reboot

PATCH In review John Stultz 7 participants 14 messages Jun 2, 2020 – Sep 7, 2020 Qualcomm Atheros ATH10K wireless driver

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 wireless: ath10k: Return early in ath10k_qmi_event_server_exit() to avoid hard crash on reboot

Ever since 5.7-rc1, if we call ath10k_qmi_remove_msa_permission(), the db845c hard crashes on reboot, resulting in the device getting stuck in the usb crash debug mode and not coming back up wihthout a hard power off.

This hack avoids the issue by returning early in
ath10k_qmi_event_server_exit().

A better solution is very much desired!

Feedback and suggestions welcome!

Cc Rakesh Pillai <[email protected]>
Cc Govind Singh <[email protected]>
Cc Bjorn Andersson <[email protected]>
Cc Niklas Cassel <[email protected]>
Cc Manivannan Sadhasivam <[email protected]>
Cc Amit Pundir <[email protected]>
Cc Brian Norris <[email protected]>
Cc Kalle Valo <[email protected]>
---
 drivers/net/wireless/ath/ath10k/qmi.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/net/wireless/ath/ath10k/qmi.c +5 -0
index 85dce43c5439..ab38562ce1cb 100644
@@ -854,6 +854,11 @@ static void ath10k_qmi_event_server_exit(struct ath10k_qmi *qmi)
 	struct ath10k *ar = qmi->ar;
 	struct ath10k_snoc *ar_snoc = ath10k_snoc_priv(ar);
 
+	/*
+	 * HACK: Calling ath10k_qmi_remove_msa_permission causes
+	 * hardware to hard crash on reboot
+	 */
+	return;
 	ath10k_qmi_remove_msa_permission(qmi);
 	ath10k_core_free_board_files(ar);
 	if (!test_bit(ATH10K_SNOC_FLAG_UNREGISTERING, &ar_snoc->flags))
Signature · 1 line
2.17.1
#1 lore raw

+ Sibi

Quoted 10 lines · John Stultz

Ever since 5.7-rc1, if we call ath10k_qmi_remove_msa_permission(), the db845c hard crashes on reboot, resulting in the device getting stuck in the usb crash debug mode and not coming back up wihthout a hard power off.

This hack avoids the issue by returning early in
ath10k_qmi_event_server_exit().

A better solution is very much desired!

Any chance you can bisect what caused this? There are a lot of non-ath10k pieces involved in this stuff.

Brian

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#2 lore raw
Quoted 10 lines · Brian Norris

+ Sibi

Quoted 6 lines · John Stultz

Ever since 5.7-rc1, if we call ath10k_qmi_remove_msa_permission(), the db845c hard crashes on reboot, resulting in the device getting stuck in the usb crash debug mode and not coming back up wihthout a hard power off.

Quoted 7 lines
Quoted 4 lines

This hack avoids the issue by returning early in
ath10k_qmi_event_server_exit().

A better solution is very much desired!

Any chance you can bisect what caused this? There are a lot of non-ath10k pieces involved in this stuff.

Amit had spent some work on chasing it down to the in kernel qrtr-ns work, and reported it here:
https://lists.infradead.org/pipermail/ath10k/2020-April/014970.html

But that discussion seemingly stalled out, so I came up with this hack to workaround it for us.

thanks
-john

#3 lore raw
Quoted 16 lines · John Stultz
Quoted 7 lines · Brian Norris
Quoted 6 lines · John Stultz

Ever since 5.7-rc1, if we call ath10k_qmi_remove_msa_permission(), the db845c hard crashes on reboot, resulting in the device getting stuck in the usb crash debug mode and not coming back up wihthout a hard power off.

Quoted 7 lines
Quoted 4 lines

This hack avoids the issue by returning early in
ath10k_qmi_event_server_exit().

A better solution is very much desired!

Any chance you can bisect what caused this? There are a lot of non-ath10k pieces involved in this stuff.

Quoted 6 lines

Amit had spent some work on chasing it down to the in kernel qrtr-ns work, and reported it here:
https://lists.infradead.org/pipermail/ath10k/2020-April/014970.html

But that discussion seemingly stalled out, so I came up with this hack to workaround it for us.

If I'm reading it right, then that means we should revert this stuff from v5.7-rc1:

0c2204a4ad71 net: qrtr: Migrate nameservice to kernel from userspace

At least, until people can resolve the tail end of that thread. New features (ath11k, etc.) are not a reason to break existing features (ath10k/wcn3990).

Brian

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#4 lore raw
Quoted 24 lines · Brian Norris
Quoted 16 lines · John Stultz
Quoted 7 lines · Brian Norris
Quoted 6 lines · John Stultz

Ever since 5.7-rc1, if we call ath10k_qmi_remove_msa_permission(), the db845c hard crashes on reboot, resulting in the device getting stuck in the usb crash debug mode and not coming back up wihthout a hard power off.

Quoted 7 lines
Quoted 4 lines

This hack avoids the issue by returning early in
ath10k_qmi_event_server_exit().

A better solution is very much desired!

Any chance you can bisect what caused this? There are a lot of non-ath10k pieces involved in this stuff.

Quoted 6 lines

Amit had spent some work on chasing it down to the in kernel qrtr-ns work, and reported it here:
https://lists.infradead.org/pipermail/ath10k/2020-April/014970.html

But that discussion seemingly stalled out, so I came up with this hack to workaround it for us.

Quoted 8 lines

If I'm reading it right, then that means we should revert this stuff from v5.7-rc1:

0c2204a4ad71 net: qrtr: Migrate nameservice to kernel from userspace

At least, until people can resolve the tail end of that thread. New features (ath11k, etc.) are not a reason to break existing features (ath10k/wcn3990).

I don't agree with this. If you read through the replies to the bug report, it is clear that NS migration uncovered a corner case or even a bug. So we should try to fix that indeed.

Govind: Did you get chance to work on fixing this issue?

Thanks,
Mani

Quoted 2 lines

Brian

#5 lore raw

Hi Mani,

Quoted 34 lines · Manivannan Sadhasivam
Quoted 24 lines · Brian Norris
Quoted 16 lines · John Stultz
Quoted 7 lines · Brian Norris
Quoted 6 lines · John Stultz

Ever since 5.7-rc1, if we call ath10k_qmi_remove_msa_permission(), the db845c hard crashes on reboot, resulting in the device getting stuck in the usb crash debug mode and not coming back up wihthout a hard power off.

Quoted 7 lines
Quoted 4 lines

This hack avoids the issue by returning early in
ath10k_qmi_event_server_exit().

A better solution is very much desired!

Any chance you can bisect what caused this? There are a lot of non-ath10k pieces involved in this stuff.

Quoted 6 lines

Amit had spent some work on chasing it down to the in kernel qrtr-ns work, and reported it here:
https://lists.infradead.org/pipermail/ath10k/2020-April/014970.html

But that discussion seemingly stalled out, so I came up with this hack to workaround it for us.

Quoted 8 lines

If I'm reading it right, then that means we should revert this stuff from v5.7-rc1:

0c2204a4ad71 net: qrtr: Migrate nameservice to kernel from userspace

At least, until people can resolve the tail end of that thread. New features (ath11k, etc.) are not a reason to break existing features (ath10k/wcn3990).

Quoted 6 lines

I don't agree with this. If you read through the replies to the bug report, it is clear that NS migration uncovered a corner case or even a bug. So we should try to fix that indeed.

Govind: Did you get chance to work on fixing this issue?

I have done basic testing by moving msa map/unmap from qmi service callbacks to init/de-init path. I will send patch for review.
Reason for del_server needs to investigated from rproc side.

Quoted 5 lines

Thanks,
Mani

Quoted 2 lines

Brian

Thanks,
Govind

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#6 lore raw
Quoted 41 lines · On 2020-06-03 15:37, [email protected] wrote:

Hi Mani,

Quoted 38 lines · Manivannan Sadhasivam
Quoted 24 lines · Brian Norris
Quoted 16 lines · John Stultz
Quoted 7 lines · Brian Norris
Quoted 6 lines · John Stultz

Ever since 5.7-rc1, if we call ath10k_qmi_remove_msa_permission(), the db845c hard crashes on reboot, resulting in the device getting stuck in the usb crash debug mode and not coming back up wihthout a hard power off.

Quoted 7 lines
Quoted 4 lines

This hack avoids the issue by returning early in
ath10k_qmi_event_server_exit().

A better solution is very much desired!

Any chance you can bisect what caused this? There are a lot of non-ath10k pieces involved in this stuff.

Quoted 6 lines

Amit had spent some work on chasing it down to the in kernel qrtr-ns work, and reported it here:
https://lists.infradead.org/pipermail/ath10k/2020-April/014970.html

But that discussion seemingly stalled out, so I came up with this hack to workaround it for us.

Quoted 8 lines

If I'm reading it right, then that means we should revert this stuff from v5.7-rc1:

0c2204a4ad71 net: qrtr: Migrate nameservice to kernel from userspace

At least, until people can resolve the tail end of that thread. New features (ath11k, etc.) are not a reason to break existing features (ath10k/wcn3990).

I don't agree with this. If you read through the replies to the bug report, it is clear that NS migration uncovered a corner case or even a bug. So we should try to fix that indeed.

Quoted 7 lines
Quoted 2 lines

Govind: Did you get chance to work on fixing this issue?

I have done basic testing by moving msa map/unmap from qmi service callbacks to init/de-init path.
I will send patch for review.
Reason for del_server needs to investigated from rproc side.

Govind,
On receiving SIGTERM, rmtfs would try
to perform a graceful shutdown of the
modem, that should be the source of
the del_server.

Quoted 9 lines
Quoted 6 lines

Thanks,
Mani

Quoted 2 lines

Brian

Thanks,
Govind

Signature · 7 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project.

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#7 lore raw
In reply to Manivannan Sadhasivam
Quoted 30 lines · Manivannan Sadhasivam
Quoted 29 lines · Brian Norris
Quoted 16 lines · John Stultz
Quoted 7 lines · Brian Norris
Quoted 6 lines · John Stultz

Ever since 5.7-rc1, if we call ath10k_qmi_remove_msa_permission(), the db845c hard crashes on reboot, resulting in the device getting stuck in the usb crash debug mode and not coming back up wihthout a hard power off.

Quoted 7 lines
Quoted 4 lines

This hack avoids the issue by returning early in
ath10k_qmi_event_server_exit().

A better solution is very much desired!

Any chance you can bisect what caused this? There are a lot of non-ath10k pieces involved in this stuff.

Quoted 6 lines

Amit had spent some work on chasing it down to the in kernel qrtr-ns work, and reported it here:
https://lists.infradead.org/pipermail/ath10k/2020-April/014970.html

But that discussion seemingly stalled out, so I came up with this hack to workaround it for us.

If I'm reading it right, then that means we should revert this stuff from v5.7-rc1:

0c2204a4ad71 net: qrtr: Migrate nameservice to kernel from userspace

Quoted 7 lines
Quoted 3 lines

At least, until people can resolve the tail end of that thread. New features (ath11k, etc.) are not a reason to break existing features (ath10k/wcn3990).

I don't agree with this. If you read through the replies to the bug report, it is clear that NS migration uncovered a corner case or even a bug. So we should try to fix that indeed.

I'm with Mani, we should try to fix ath10k instead. Hopefully we can find a fix soon.

Forcing QCA6390 users to use the userspace qrtr-ns would be bad user experience, I really would want to avoid that.

Signature · 6 lines
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#8 lore raw
Quoted 6 lines · Kalle Valo
Quoted 3 lines

I don't agree with this. If you read through the replies to the bug report, it is clear that NS migration uncovered a corner case or even a bug. So we should try to fix that indeed.

I'm with Mani, we should try to fix ath10k instead. Hopefully we can find a fix soon.

Hi Team,

Any updates on this? I can reproduce this hard crash on v5.9-rc1 as well.

It is not a blocker for us because we switched to a userspace workaround, where we do not wait for modem to shutdown gracefully and SIGKILL it instead, during the shutdown/reboot process. But I'm happy to take a swing at any intermediate/in-progress solution available.

Regards,
Amit Pundir

Quoted 6 lines

Forcing QCA6390 users to use the userspace qrtr-ns would be bad user experience, I really would want to avoid that.

--
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

#9 lore raw
Quoted 8 lines · Amit Pundir
Quoted 6 lines · Kalle Valo
Quoted 3 lines

I don't agree with this. If you read through the replies to the bug report, it is clear that NS migration uncovered a corner case or even a bug. So we should try to fix that indeed.

I'm with Mani, we should try to fix ath10k instead. Hopefully we can find a fix soon.

Quoted 8 lines

Hi Team,

Any updates on this? I can reproduce this hard crash on v5.9-rc1 as well.

It is not a blocker for us because we switched to a userspace workaround, where we do not wait for modem to shutdown gracefully and SIGKILL it instead, during the shutdown/reboot process. But I'm happy to take a swing at any intermediate/in-progress solution available.

Govind submitted this patch and later he asked to drop it, but I think it would be a good idea to test it anyway:

ath10k: Move msa region map/unmap to init/deinit path

https://lkml.kernel.org/r/[email protected]

(patchwork is down so I cannot give a patchwork link)

Signature · 6 lines
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#10 lore raw

Hi Kalle,

Quoted 19 lines · Kalle Valo
Quoted 8 lines · Amit Pundir
Quoted 6 lines · Kalle Valo
Quoted 3 lines

I don't agree with this. If you read through the replies to the bug report, it is clear that NS migration uncovered a corner case or even a bug. So we should try to fix that indeed.

I'm with Mani, we should try to fix ath10k instead. Hopefully we can find a fix soon.

Quoted 8 lines

Hi Team,

Any updates on this? I can reproduce this hard crash on v5.9-rc1 as well.

It is not a blocker for us because we switched to a userspace workaround, where we do not wait for modem to shutdown gracefully and SIGKILL it instead, during the shutdown/reboot process. But I'm happy to take a swing at any intermediate/in-progress solution available.

Quoted 8 lines

Govind submitted this patch and later he asked to drop it, but I think it would be a good idea to test it anyway:

ath10k: Move msa region map/unmap to init/deinit path

https://lkml.kernel.org/r/[email protected]

(patchwork is down so I cannot give a patchwork link)

This patchwork is not fixing the issue and changing MSA mapping sequence is major design change.
This issue is only seen with DB845 which uses SCM call, newer targets QCS404/SC7180/SM8150 will not have this issue as MSA mapping is hard-coded in TZ. Probably changes in qmi layer to give different indication for this scenario and changes in FW is required to mitigate this issue gracefully.

BR,
Govind

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#11 lore raw
Quoted 28 lines · Govind Singh
Quoted 27 lines · Kalle Valo
Quoted 10 lines · Amit Pundir
Quoted 6 lines · Kalle Valo
Quoted 3 lines

I don't agree with this. If you read through the replies to the bug report, it is clear that NS migration uncovered a corner case or even a bug. So we should try to fix that indeed.

I'm with Mani, we should try to fix ath10k instead. Hopefully we can find a fix soon.

Hi Team,

Quoted 7 lines

Any updates on this? I can reproduce this hard crash on v5.9-rc1 as well.

It is not a blocker for us because we switched to a userspace workaround, where we do not wait for modem to shutdown gracefully and SIGKILL it instead, during the shutdown/reboot process. But I'm happy to take a swing at any intermediate/in-progress solution available.

Govind submitted this patch and later he asked to drop it, but I think it would be a good idea to test it anyway:

ath10k: Move msa region map/unmap to init/deinit path

https://lkml.kernel.org/r/[email protected]

Quoted 8 lines
Quoted 1 line

(patchwork is down so I cannot give a patchwork link)

This patchwork is not fixing the issue and changing MSA mapping sequence is major design change. This issue is only seen with DB845 which uses SCM call, newer targets QCS404/SC7180/SM8150 will not have this issue as MSA mapping is hard-coded in TZ. Probably changes in qmi layer to give different indication for this scenario and changes in FW is required to mitigate this issue gracefully.

Oh, bad news :/ Can anyone look at that in detail? Even a quick hack patch would get this forward.

Signature · 6 lines
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#12 lore raw
In reply to Govind Singh
Quoted 28 lines · Govind Singh
Quoted 27 lines · Kalle Valo
Quoted 10 lines · Amit Pundir
Quoted 6 lines · Kalle Valo
Quoted 3 lines

I don't agree with this. If you read through the replies to the bug report, it is clear that NS migration uncovered a corner case or even a bug. So we should try to fix that indeed.

I'm with Mani, we should try to fix ath10k instead. Hopefully we can find a fix soon.

Hi Team,

Quoted 7 lines

Any updates on this? I can reproduce this hard crash on v5.9-rc1 as well.

It is not a blocker for us because we switched to a userspace workaround, where we do not wait for modem to shutdown gracefully and SIGKILL it instead, during the shutdown/reboot process. But I'm happy to take a swing at any intermediate/in-progress solution available.

Govind submitted this patch and later he asked to drop it, but I think it would be a good idea to test it anyway:

ath10k: Move msa region map/unmap to init/deinit path

https://lkml.kernel.org/r/[email protected]

Quoted 8 lines
Quoted 1 line

(patchwork is down so I cannot give a patchwork link)

This patchwork is not fixing the issue and changing MSA mapping sequence is major design change. This issue is only seen with DB845 which uses SCM call, newer targets QCS404/SC7180/SM8150 will not have this issue as MSA mapping is hard-coded in TZ. Probably changes in qmi layer to give different indication for this scenario and changes in FW is required to mitigate this issue gracefully.

Oh, bad news :/ Can anyone look at that in detail? Even a quick hack patch would get this forward.

Signature · 1 line
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
#13 lore raw
In reply to John Stultz
Quoted 12 lines · John Stultz

Ever since 5.7-rc1, if we call ath10k_qmi_remove_msa_permission(), the db845c hard crashes on reboot, resulting in the device getting stuck in the usb crash debug mode and not coming back up wihthout a hard power off.

This hack avoids the issue by returning early in
ath10k_qmi_event_server_exit().

A better solution is very much desired!

Feedback and suggestions welcome!

Quoted 11 lines
Cc Rakesh Pillai <[email protected]>
Cc Govind Singh <[email protected]>
Cc Bjorn Andersson <[email protected]>
Cc Niklas Cassel <[email protected]>
Cc Manivannan Sadhasivam <[email protected]>
Cc Amit Pundir <[email protected]>
Cc Brian Norris <[email protected]>
Cc Kalle Valo <[email protected]>

Just so you know: as you didn't CC linux-wireless it's not on patchwork and hence not on my radar. But hopefully we find a better solution to fix this.

Signature · 1 line
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches