USB EHCI: repeated resets on full and low speed devices

With the USB 3.0/3.1 controller on MSI B450-A Pro Max motherboard, full speed and low speed devices see constant resets making keyboards and mouse unreliable and unusable. These resets are caused by detection of stall in qtd_copy_status() and returning EPROTO which in turn results in TT buffers in hub being cleared. Hubs do not seem to repsond well to this and seem to hang which causes further USB transactions to time out. A reset finally clears the issue until we repeat the cycle all over again.

Signed-off-by Khalid Aziz <[email protected]>

Cc Khalid Aziz <[email protected]>

---
 drivers/usb/host/ehci-q.c | 4 ----
 1 file changed, 4 deletions(-)

drivers/usb/host/ehci-q.c +0 -0

index 8a5c9b3ebe1e..7d4b2bc4633c 100644

Quoted 20 lines · Khalid Aziz

With the USB 3.0/3.1 controller on MSI B450-A Pro Max motherboard, full speed and low speed devices see constant resets making keyboards and mouse unreliable and unusable. These resets are caused by detection of stall in qtd_copy_status() and returning EPROTO which in turn results in TT buffers in hub being cleared. Hubs do not seem to repsond well to this and seem to hang which causes further USB transactions to time out. A reset finally clears the issue until we repeat the cycle all over again.

Signed-off-by Khalid Aziz <[email protected]>

Cc Khalid Aziz <[email protected]>

---
 drivers/usb/host/ehci-q.c | 4 ----
 1 file changed, 4 deletions(-)

drivers/usb/host/ehci-q.c +0 -0

index 8a5c9b3ebe1e..7d4b2bc4633c 100644

drivers/usb/host/ehci-q.c static int qtd_copy_status ( · line 219

Quoted 10 lines

drivers/usb/host/ehci-q.c +0 -4

@@ -214,10 +214,6 @@ static int qtd_copy_status (
 		 * When MMF is active and PID Code is IN, queue is halted.
 		 * EHCI Specification, Table 4-13.
 		 */
-		} else if ((token & QTD_STS_MMF) &&
-					(QTD_PID(token) == PID_CODE_IN)) {
-			status = -EPROTO;
-		/* CERR nonzero + halt --> stall */
 		} else if (QTD_CERR(token)) {
 			status = -EPIPE;
 

Removing this check is not a good idea, any chance you can come up with some other test instead for this broken hardware?

What about getting a USB hub that works? :)

[Resending since I screwed up linux-usb mailing list address in cut-n-paste in original email]

I recently replaced the motherboard on my desktop with an MSI B450-A Pro Max motherboard. Since then my keybaords, mouse and tablet have become very unreliable. I see messages like this over and over in dmesg:

ug 23 00:01:49 rhapsody kernel: [198769.314732] usb 1-2.4: reset full-speed USB
device number 27 using ehci-pci

Aug 23 00:01:49 rhapsody kernel: [198769.562234] usb 1-2.1: reset full-speed USB
 device number 28 using ehci-pci
Aug 23 00:01:52 rhapsody kernel: [198772.570704] usb 1-2.1: reset full-speed USB
 device number 28 using ehci-pci
Aug 23 00:02:02 rhapsody kernel: [198782.526669] usb 1-2.4: reset full-speed USB
 device number 27 using ehci-pci
Aug 23 00:02:03 rhapsody kernel: [198782.714660] usb 1-2.1: reset full-speed USB
 device number 28 using ehci-pci
Aug 23 00:02:04 rhapsody kernel: [198784.210171] usb 1-2.3: reset low-speed USB device number 26 using ehci-pci
Aug 23 00:02:06 rhapsody kernel: [198786.110181] usb 1-2.4: reset full-speed USB device number 27 using ehci-pci
Aug 23 00:02:08 rhapsody kernel: [198787.726158] usb 1-2.4: reset full-speed USB device number 27 using ehci-pci
Aug 23 00:02:10 rhapsody kernel: [198790.126628] usb 1-2.1: reset full-speed USB device number 28 using ehci-pci
Aug 23 00:02:10 rhapsody kernel: [198790.314141] usb 1-2.4: reset full-speed USB device number 27 using ehci-pci
Aug 23 00:02:12 rhapsody kernel: [198792.518765] usb 1-2.4: reset full-speed USB device number 27 using ehci-pci

The devices I am using are:

- Logitech K360 wireless keyboard
- Wired Lenovo USB keyboard
- Wired Lenovo USB mouse
- Wired Wacom Intuos tablet

After a reset, the wireless keyboard simply stops working. Rest of the devices keep seeing intermittent failure.

I tried various combinations of hubs and USB controllers to see what works. MSI B450-A motherboard has USB 3.0 and USB 3.1 controllers. I added a USB 2.0 PCI card as well for this test:

03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller (rev 01) 29:01.0 USB controller: NEC Corporation OHCI USB Controller (rev 43)
29:01.1 USB controller: NEC Corporation OHCI USB Controller (rev 43)
29:01.2 USB controller: NEC Corporation uPD72010x USB 2.0 Controller (rev 04)
2c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

I have a bus powered USB 3.0 hub, a bus powered USB 2.0 hub and a self powered USB 2.0 hub built into my monitor.

I have connected my devices directly into the ports on motherboard and PCI card as well as into external hub. Here are the results I saw when devices wee plugged into various combination of ports:

1. USB 3.0/3.1 controller - does NOT work
2. USB 2.0 controller - WORKS
3. USB 3.0/3.1 controller -> Self powered USB 2.0 hub in monitor - does NOT work
4. USB 3.0/3.1 controller -> bus powered USB 3.0 hub - does NOT work
5. USB 3.0/3.1 controller -> Bus powered USB 2.0 hub - WORKS
7. USB 2.0 controller -> Bus powered USB 3.0 hub - does NOT work
8. USB 2.0 controller -> Bus powered 2.0 hub - Does not work

I narrowed the failure down to following lines (this code was added in 5.5 with commit 64cc3f12d1c7 "USB: EHCI: Do not return -EPIPE when hub is disconnected"):

drivers/usb/host/ehci-q.c:

217 } else if ((token & QTD_STS_MMF) &&
218 (QTD_PID(token) == PID_CODE_IN)) {
219 status = -EPROTO;
220 /* CERR nonzero + halt --> stall */

At the time of failure, when we reach this conditional, token is either 0x80408d46 or 0x408d46 which means following bits are set:

QTD_STS_STS, QTD_STS_MMF, QTD_STS_HALT, QTD_IOC, QTD_TOGGLE

and

        QTD_PID = 1
        QTD_CERR = 3
        QTD_LENGTH = 0x40 (64)

This causes the branch "(token & QTD_STS_MMF) && (QTD_PID(token) == PID_CODE_IN" to be taken and qtd_copy_status() returns EPROTO. This return value in qh_completions() results in ehci_clear_tt_buffer() being called:

drivers/usb/host/ehci-q.c:
472 /* As part of low/full-speed endpoint-halt processi ng
473 * we must clear the TT buffer (11.17.5).
474 */
475 if (unlikely(last_status != -EINPROGRESS &&
476 last_status != -EREMOTEIO)) {

 477                                 /* The TT's in some hubs malfunction when t     hey
 478                                  * receive this request following a STALL (     they
 479                                  * stop sending isochronous packets).  Sinc     e a
 480                                  * STALL can't leave the TT buffer in a bus     y
 481                                  * state (if you believe Figures 11-48 - 11     -51
 482                                  * in the USB 2.0 spec), we won't clear the      TT
 483                                  * buffer in this case.  Strictly speaking      this
 484                                  * is a violation of the spec.
 485                                  */

486 if (last_status != -EPIPE)
487 ehci_clear_tt_buffer(ehci, qh, urb,
488 token);
489 }

It seems like clearing TT buffers in this case is resulting in hub hanging. A USB reset gets it going again until we repeat the cycle over again. The comment in this code says "The TT's in some hubs malfunction when they receive this request following a STALL (they stop sending isochronous packets)". That may be what is happening.

Removing the code that returns EPROTO for such case solves the problem on my machine (as in the RFC patch)

but that probably is not
the right solution. I do not understand USB protocol well enough to propose a better solution. Does anyone have a better idea?

Quoted 53 lines · Khalid Aziz

[Resending since I screwed up linux-usb mailing list address in cut-n-paste in original email]

I recently replaced the motherboard on my desktop with an MSI B450-A Pro Max motherboard. Since then my keybaords, mouse and tablet have become very unreliable. I see messages like this over and over in dmesg:

ug 23 00:01:49 rhapsody kernel: [198769.314732] usb 1-2.4: reset full-speed USB
device number 27 using ehci-pci

Aug 23 00:01:49 rhapsody kernel: [198769.562234] usb 1-2.1: reset full-speed USB
 device number 28 using ehci-pci
Aug 23 00:01:52 rhapsody kernel: [198772.570704] usb 1-2.1: reset full-speed USB
 device number 28 using ehci-pci
Aug 23 00:02:02 rhapsody kernel: [198782.526669] usb 1-2.4: reset full-speed USB
 device number 27 using ehci-pci
Aug 23 00:02:03 rhapsody kernel: [198782.714660] usb 1-2.1: reset full-speed USB
 device number 28 using ehci-pci
Aug 23 00:02:04 rhapsody kernel: [198784.210171] usb 1-2.3: reset low-speed USB device number 26 using ehci-pci
Aug 23 00:02:06 rhapsody kernel: [198786.110181] usb 1-2.4: reset full-speed USB device number 27 using ehci-pci
Aug 23 00:02:08 rhapsody kernel: [198787.726158] usb 1-2.4: reset full-speed USB device number 27 using ehci-pci
Aug 23 00:02:10 rhapsody kernel: [198790.126628] usb 1-2.1: reset full-speed USB device number 28 using ehci-pci
Aug 23 00:02:10 rhapsody kernel: [198790.314141] usb 1-2.4: reset full-speed USB device number 27 using ehci-pci
Aug 23 00:02:12 rhapsody kernel: [198792.518765] usb 1-2.4: reset full-speed USB device number 27 using ehci-pci

The devices I am using are:

- Logitech K360 wireless keyboard
- Wired Lenovo USB keyboard
- Wired Lenovo USB mouse
- Wired Wacom Intuos tablet

After a reset, the wireless keyboard simply stops working. Rest of the devices keep seeing intermittent failure.

I tried various combinations of hubs and USB controllers to see what works. MSI B450-A motherboard has USB 3.0 and USB 3.1 controllers. I added a USB 2.0 PCI card as well for this test:

03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller (rev 01) 29:01.0 USB controller: NEC Corporation OHCI USB Controller (rev 43)
29:01.1 USB controller: NEC Corporation OHCI USB Controller (rev 43)
29:01.2 USB controller: NEC Corporation uPD72010x USB 2.0 Controller (rev 04)
2c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

I have a bus powered USB 3.0 hub, a bus powered USB 2.0 hub and a self powered USB 2.0 hub built into my monitor.

I have connected my devices directly into the ports on motherboard and PCI card as well as into external hub. Here are the results I saw when devices wee plugged into various combination of ports:

Quoted 8 lines

1. USB 3.0/3.1 controller - does NOT work
2. USB 2.0 controller - WORKS
3. USB 3.0/3.1 controller -> Self powered USB 2.0 hub in monitor - does NOT work
4. USB 3.0/3.1 controller -> bus powered USB 3.0 hub - does NOT work
5. USB 3.0/3.1 controller -> Bus powered USB 2.0 hub - WORKS
7. USB 2.0 controller -> Bus powered USB 3.0 hub - does NOT work
8. USB 2.0 controller -> Bus powered 2.0 hub - Does not work

The error messages in your log extract all refer to ehci-pci, which is the driver for a USB-2 controller. They are completely unrelated to any problems you may be having with USB-3 controllers.

Quoted 48 lines

I narrowed the failure down to following lines (this code was added in 5.5 with commit 64cc3f12d1c7 "USB: EHCI: Do not return -EPIPE when hub is disconnected"):

drivers/usb/host/ehci-q.c:

217 } else if ((token & QTD_STS_MMF) &&
218 (QTD_PID(token) == PID_CODE_IN)) {
219 status = -EPROTO;
220 /* CERR nonzero + halt --> stall */

At the time of failure, when we reach this conditional, token is either 0x80408d46 or 0x408d46 which means following bits are set:

QTD_STS_STS, QTD_STS_MMF, QTD_STS_HALT, QTD_IOC, QTD_TOGGLE

and

        QTD_PID = 1
        QTD_CERR = 3
        QTD_LENGTH = 0x40 (64)

This causes the branch "(token & QTD_STS_MMF) && (QTD_PID(token) == PID_CODE_IN" to be taken and qtd_copy_status() returns EPROTO. This return value in qh_completions() results in ehci_clear_tt_buffer() being called:

drivers/usb/host/ehci-q.c:
472 /* As part of low/full-speed endpoint-halt processi ng
473 * we must clear the TT buffer (11.17.5).
474 */
475 if (unlikely(last_status != -EINPROGRESS &&
476 last_status != -EREMOTEIO)) {

 477                                 /* The TT's in some hubs malfunction when t     hey
 478                                  * receive this request following a STALL (     they
 479                                  * stop sending isochronous packets).  Sinc     e a
 480                                  * STALL can't leave the TT buffer in a bus     y
 481                                  * state (if you believe Figures 11-48 - 11     -51
 482                                  * in the USB 2.0 spec), we won't clear the      TT
 483                                  * buffer in this case.  Strictly speaking      this
 484                                  * is a violation of the spec.
 485                                  */

486 if (last_status != -EPIPE)
487 ehci_clear_tt_buffer(ehci, qh, urb,
488 token);
489 }

Quoted 5 lines

It seems like clearing TT buffers in this case is resulting in hub hanging. A USB reset gets it going again until we repeat the cycle over again. The comment in this code says "The TT's in some hubs malfunction when they receive this request following a STALL (they stop sending isochronous packets)". That may be what is happening.

What makes you think that? Do you have any evidence that the hub is receiving a STALL? Indeed, the commit you referenced above specifically mentions that when MMF is set and the PID code is IN then it is not a STALL.

Quoted 2 lines

Removing the code that returns EPROTO for such case solves the problem on my machine (as in the RFC patch)

It certainly can't solve the problem for any USB-3 connections, because the patch doesn't touch any of the USB-3 driver code.

Quoted 4 lines

but that probably is not
the right solution. I do not understand USB protocol well enough to propose a better solution. Does anyone have a better idea?

Can you collect a usbmon trace showing an example of this problem?

One possibility is to introduce a special quirk for the NEC uPD72010x EHCI controller. But we should hold off on that until we know exactly what is happening.

Alan Stern

Quoted 7 lines

Quoted 7 lines

At the time of failure, when we reach this conditional, token is either 0x80408d46 or 0x408d46 which means following bits are set:

QTD_STS_STS, QTD_STS_MMF, QTD_STS_HALT, QTD_IOC, QTD_TOGGLE

and

Quoted 8 lines

Quoted 8 lines

        QTD_PID = 1
        QTD_CERR = 3
        QTD_LENGTH = 0x40 (64)

This causes the branch "(token & QTD_STS_MMF) && (QTD_PID(token) == PID_CODE_IN" to be taken and qtd_copy_status() returns EPROTO. This return value in qh_completions() results in ehci_clear_tt_buffer() being called:

Quoted 2 lines · Alan Stern

Can you collect a usbmon trace showing an example of this problem?

I have attached usbmon traces for when USB hub with keyboards and mouse is plugged into USB 2.0 port and when it is plugged into the NEC USB 3.0 port.

Quoted 4 lines · Khalid Aziz

Quoted 2 lines · Alan Stern

Can you collect a usbmon trace showing an example of this problem?

Quoted 3 lines

I have attached usbmon traces for when USB hub with keyboards and mouse is plugged into USB 2.0 port and when it is plugged into the NEC USB 3.0 port.

The usbmon traces show lots of errors, but no Clear-TT events. The large number of errors suggests that you've got a hardware problem; either a bad hub or bad USB connections.

Quoted 5 lines · Alan Stern

Quoted 4 lines · Khalid Aziz

Quoted 2 lines · Alan Stern

Can you collect a usbmon trace showing an example of this problem?

Quoted 7 lines

Quoted 3 lines

I have attached usbmon traces for when USB hub with keyboards and mouse is plugged into USB 2.0 port and when it is plugged into the NEC USB 3.0 port.

The usbmon traces show lots of errors, but no Clear-TT events. The large number of errors suggests that you've got a hardware problem; either a bad hub or bad USB connections.

That is what I thought initially which is why I got additional hubs and a USB 2.0 PCI card to test. I am seeing errors across 3 USB controllers, 4 USB hubs and 4 slow/full speed devices. All of the hubs and slow/full devices work with zero errors on my laptop. My keyboard/mouse devices and 2 of my USB hubs predate motherboard update and they all worked flawlessly before the motherboard upgrade. Some combinations of these also works with no errors on my desktop with new motherboard that I had listed in my original email:

2. USB 2.0 controller - WORKS
5. USB 3.0/3.1 controller -> Bus powered USB 2.0 hub - WORKS

I am not seeing a common failure here that would point to any specific hardware being bad. Besides, that one code change (which I still can't say is the right code change) in ehci-q.c makes USB 2.0 controller work reliably with all my devices.

Quoted 14 lines · Khalid Aziz

Quoted 5 lines · Alan Stern

Quoted 4 lines · Khalid Aziz

Quoted 2 lines · Alan Stern

Can you collect a usbmon trace showing an example of this problem?

Quoted 7 lines

Quoted 3 lines

I have attached usbmon traces for when USB hub with keyboards and mouse is plugged into USB 2.0 port and when it is plugged into the NEC USB 3.0 port.

The usbmon traces show lots of errors, but no Clear-TT events. The large number of errors suggests that you've got a hardware problem; either a bad hub or bad USB connections.

Quoted 8 lines

That is what I thought initially which is why I got additional hubs and a USB 2.0 PCI card to test. I am seeing errors across 3 USB controllers, 4 USB hubs and 4 slow/full speed devices. All of the hubs and slow/full devices work with zero errors on my laptop. My keyboard/mouse devices and 2 of my USB hubs predate motherboard update and they all worked flawlessly before the motherboard upgrade. Some combinations of these also works with no errors on my desktop with new motherboard that I had listed in my original email:

It's a very puzzling situation.

One thing which probably would work well, surprisingly, would be to buy an old USB-1.1 hub and plug it into the PCI card. That combination is likely to be similar to what you see when plugging the devices directly into the PCI card. It might even work okay with the USB-3 controllers.

Quoted 7 lines

2. USB 2.0 controller - WORKS
5. USB 3.0/3.1 controller -> Bus powered USB 2.0 hub - WORKS

I am not seeing a common failure here that would point to any specific hardware being bad. Besides, that one code change (which I still can't say is the right code change) in ehci-q.c makes USB 2.0 controller work reliably with all my devices.

The USB and EHCI designs are flawed in that under the circumstances you're seeing, they don't have any way to tell the difference between a STALL and a host timing error. The current code treats these situations as timing/transmission errors (resulting in device resets); your change causes them to be treated as STALLs. However, there are known, common situations in which those same symptoms really are caused by transmission errors, so we don't want to start treating them as STALLs.

Besides, I suspect that your code change does _not_ make the USB-2 controller work reliably with your devices. You should collect a usbmon trace under those conditions; I predict it will be full of STALLs. And furthermore, I believe these STALLs will not show up in a usbmon trace made with the devices plugged directly into the PCI card. If I'm right about these things, the errors are still present even with your patch; all it does is hide them.

Short of a USB bus analyzer, however, there's no way to tell what's really going on.

I have managed to find a hardware combination that seems to work, so for now at least my machine is usable. I will figure out how to interpret usbmon output and run more experiments. There seems to be a real problem in the driver somewhere and should be solved.