Skip to contentSkip to conversation

xhci: guard accesses to ep_state in xhci_endpoint_reset()

PATCH In review Phil Elwell 3 participants 4 messages Aug 31, 2021 – Sep 1, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xhci: guard accesses to ep_state in xhci_endpoint_reset()

From: Jonathan Bell <[email protected]>

See https://github.com/raspberrypi/linux/issues/3981

Two read-modify-write cycles on ep->ep_state are not guarded by xhci->lock. Fix these.

---
 drivers/usb/host/xhci.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/usb/host/xhci.c +5 -0
index f3dabd02382c2..902f410874e8e 100644
@@ -3213,10 +3213,13 @@ static void xhci_endpoint_reset(struct usb_hcd *hcd,
 		return;
 
 	/* Bail out if toggle is already being cleared by a endpoint reset */
+	spin_lock_irqsave(&xhci->lock, flags);
 	if (ep->ep_state & EP_HARD_CLEAR_TOGGLE) {
 		ep->ep_state &= ~EP_HARD_CLEAR_TOGGLE;
+		spin_unlock_irqrestore(&xhci->lock, flags);
 		return;
 	}
+	spin_unlock_irqrestore(&xhci->lock, flags);
 	/* Only interrupt and bulk ep's use data toggle, USB2 spec 5.5.4-> */
 	if (usb_endpoint_xfer_control(&host_ep->desc) ||
 	    usb_endpoint_xfer_isoc(&host_ep->desc))
@@ -3302,8 +3305,10 @@ static void xhci_endpoint_reset(struct usb_hcd *hcd,
 	xhci_free_command(xhci, cfg_cmd);
 cleanup:
 	xhci_free_command(xhci, stop_cmd);
+	spin_lock_irqsave(&xhci->lock, flags);
 	if (ep->ep_state & EP_SOFT_CLEAR_TOGGLE)
 		ep->ep_state &= ~EP_SOFT_CLEAR_TOGGLE;
+	spin_unlock_irqrestore(&xhci->lock, flags);
 }
 
 static int xhci_check_streams_endpoint(struct xhci_hcd *xhci,
Signature · 1 line
2.25.1
#1 lore raw
Quoted 3 lines · Phil Elwell

From: Jonathan Bell <[email protected]>

See https://github.com/raspberrypi/linux/issues/3981

Thanks, so in a nutshell the issue looks something like:

[827586.220071] xhci_hcd 0000:01:00.0: WARN Cannot submit Set TR Deq Ptr
[827586.220087] xhci_hcd 0000:01:00.0: A Set TR Deq Ptr command is pending.
[827723.160680] INFO: task usb-storage:93 blocked for more than 122 seconds.

The blocked task is probably because xhci driver failed to give back the URB after failing to submit a "Set TR Deq Ptr" command. This part should be fixed in:
https://lore.kernel.org/r/[email protected]
which is currently in usb-next, and should be in 5.15-rc1 and future 5.12+ stable.

Quoted 4 lines

Two read-modify-write cycles on ep->ep_state are not guarded by xhci->lock. Fix these.

This is probably one cause for the "Warn Cannot submit Set TR Deq Ptr A Set TR Deq Ptr command is pending" message.
Another possibility is that with UAS and streams we have several transfer rings per endpoint, meaning that if two TDs on separate stream rings on the same endpoint both stall, or are cancelled we could see this message.

The SET_DEQ_PENDING flag in ep->ep_state should probably be per ring, not per endpoint. Then we also need a "rings_with_pending_set_deq" counter per endpoint to keep track when all set_tr_deq commands complete, and we can restart the endpoint

Anyway, my patch linked above together with this patch should make these errors a lot more harmless.

Let me know if you can trigger the issue with both these patches applied.

I'll add your patch to the queue as well.

Thanks
-Mathias

#2 lore raw

Hi Mathias,

Quoted 28 lines · Mathias Nyman
Quoted 3 lines · Phil Elwell

From: Jonathan Bell <[email protected]>

See https://github.com/raspberrypi/linux/issues/3981

Thanks, so in a nutshell the issue looks something like:

[827586.220071] xhci_hcd 0000:01:00.0: WARN Cannot submit Set TR Deq Ptr
[827586.220087] xhci_hcd 0000:01:00.0: A Set TR Deq Ptr command is pending.
[827723.160680] INFO: task usb-storage:93 blocked for more than 122 seconds.

The blocked task is probably because xhci driver failed to give back the URB after failing to submit a "Set TR Deq Ptr" command. This part should be fixed in:
https://lore.kernel.org/r/[email protected]
which is currently in usb-next, and should be in 5.15-rc1 and future 5.12+ stable.

Quoted 4 lines

Two read-modify-write cycles on ep->ep_state are not guarded by xhci->lock. Fix these.

This is probably one cause for the "Warn Cannot submit Set TR Deq Ptr A Set TR Deq Ptr command is pending" message.
Another possibility is that with UAS and streams we have several transfer rings per endpoint, meaning that if two TDs on separate stream rings on the same endpoint both stall, or are cancelled we could see this message.

Quoted 3 lines

The SET_DEQ_PENDING flag in ep->ep_state should probably be per ring, not per endpoint. Then we also need a "rings_with_pending_set_deq" counter per endpoint to keep track when all set_tr_deq commands complete, and we can restart the endpoint

Jonathan, the author of the patch, may give some detailed feedback on these statements when he has a moment - "Well, sort of... it's complicated" was the summary.

Quoted 2 lines

Anyway, my patch linked above together with this patch should make these errors a lot more harmless.

Yes, I think that's true. We have a downstream patch to warn about a pending
Set TR Deq Ptr command but proceed anyway, allowing systems to recover, but with the additional spin lock usage our users are reporting no failures _and_ no warnings.

Quoted 1 line

Let me know if you can trigger the issue with both these patches applied.

We've not tried your patch yet.

Quoted 1 line

I'll add your patch to the queue as well.

Many thanks,

Phil

#3 lore raw
Quoted 32 lines · Phil Elwell

Hi Mathias,

Quoted 28 lines · Mathias Nyman
Quoted 3 lines · Phil Elwell

From: Jonathan Bell <[email protected]>

See https://github.com/raspberrypi/linux/issues/3981

Thanks, so in a nutshell the issue looks something like:

[827586.220071] xhci_hcd 0000:01:00.0: WARN Cannot submit Set TR Deq Ptr
[827586.220087] xhci_hcd 0000:01:00.0: A Set TR Deq Ptr command is pending.
[827723.160680] INFO: task usb-storage:93 blocked for more than 122 seconds.

The blocked task is probably because xhci driver failed to give back the URB after failing to submit a "Set TR Deq Ptr" command. This part should be fixed in:
https://lore.kernel.org/r/[email protected]
which is currently in usb-next, and should be in 5.15-rc1 and future 5.12+ stable.

Quoted 4 lines

Two read-modify-write cycles on ep->ep_state are not guarded by xhci->lock. Fix these.

This is probably one cause for the "Warn Cannot submit Set TR Deq Ptr A Set TR Deq Ptr command is pending" message.
Another possibility is that with UAS and streams we have several transfer rings per endpoint, meaning that if two TDs on separate stream rings on the same endpoint both stall, or are cancelled we could see this message.

Quoted 7 lines
Quoted 3 lines

The SET_DEQ_PENDING flag in ep->ep_state should probably be per ring, not per endpoint. Then we also need a "rings_with_pending_set_deq" counter per endpoint to keep track when all set_tr_deq commands complete, and we can restart the endpoint

Jonathan, the author of the patch, may give some detailed feedback on these statements when he has a moment - "Well, sort of... it's complicated" was the summary.

The bug in this case was apparent only on usb-storage devices, so no streams. The testcase in the linked github issue repeatedly invoked smartctl on a device that didn't
support that particular ATA command so stalled the endpoint.

The unguarded read-modify-write raced with the completion of the Set TR Deq command
being executed by the controller, and the SET_DEQ_PENDING flag would get overwritten
with an invalid '1'.

I've not investigated how the xhci driver responds to stalls on stream endpoints in detail
as I've not seen any user reports of uas active and this error message appearing, but in
a wider context, getting a stall on a streaming endpoint on a Pi 4 seems to be generally
fatal until the device receives a reset and I've not determined which of the HCD, controller,
or device is stuck.

Thanks
Jonathan