Skip to contentSkip to conversation

qmi-wwan bug

DISCUSSION Shawn J. Goff 3 participants 6 messages Oct 24, 2012 – Oct 25, 2012

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I've backported qmi-wwan to 2.6.39 (it's here: https://bitbucket.org/accelecon/linux-at91/changesets), and it mostly works, but I've come across a problem. The modem will sometimes stop responding to any qmi data (but the AT commands on the TTY ports keep working). This only happens when there is significant traffic flowing through the device (downloading a large file) while at the same time, AT commands are sent to one of the TTY ports (I first noticed with my own modem query program, but I can reproduce it using microcom to send "ATI\r" in a loop). I see this problem with different devices from different manufacturers. I've only made it happen on my kernel - I tried on 3.6.2, but it seems to not happen there. I've also tried using a similar modem that uses a different driver (sierra-net) and that doesn't have the same problem.

When it is in failure, if I try to ping an address, the system sends out several an ARP requests but gets no response. To get the device to respond again, I have to administratively set the wwan interface down, then up, use libqmi to get the connection going again, then dhcp to get an address.

I also have some USB traces of the failure and recovery process. I'm not familiar with CDC or QMI, so it's not yet clear to me exactly what's happening, but it looks like the modem just stops sending anything on its QMI endpoint for no reason.

How might I dive further into the issue? So far, my next step is to look into CDC and QMI and try to decypher the USB traces. If anyone is interested, I can share a tcpdump or a USB trace.

Signature · 1 line
Shawn J. Goff | Accelerated Concepts | Embedded Systems Engineer | 1-813-983-7501
#1 lore raw
Quoted 10 lines · Shawn J. Goff

I've backported qmi-wwan to 2.6.39 (it's here:
https://bitbucket.org/accelecon/linux-at91/changesets), and it mostly works, but I've come across a problem. The modem will sometimes stop responding to any qmi data (but the AT commands on the TTY ports keep working). This only happens when there is significant traffic flowing through the device (downloading a large file) while at the same time, AT
commands are sent to one of the TTY ports (I first noticed with my own modem query program, but I can reproduce it using microcom to send "ATI\r" in a loop).

Using the tty ports should be completely independent of any qmi activity from the host perspective. I am tempted to claim this indicates a firmware bug.

Quoted 2 lines

I see this problem with different devices from
different manufacturers.

Which still may use pretty much the same firmware, although a little less likely.

Quoted 3 lines

I've only made it happen on my kernel - I
tried
on 3.6.2, but it seems to not happen there.

Good. But I have a feeling that you switched more than just the kernel. Do you see the issue if you run your backport on the same hardware you tested 3.6.2 on?

Quoted 4 lines

I've also tried using a
similar modem that uses a different driver (sierra-net) and that doesn't
have the same problem.

Well, that is an entirely different firmware application and driver, even if the hardware is similar or even identical.

Quoted 7 lines

When it is in failure, if I try to ping an address, the system sends out
several an ARP requests but gets no response. To get the device to respond again, I have to administratively set the wwan interface down, then up, use libqmi to get the connection going again, then dhcp to get

an address.

Which sounds like the connection died. Does QMI work at this point, or is that dead too?

Quoted 10 lines

I also have some USB traces of the failure and recovery process. I'm not
familiar with CDC or QMI, so it's not yet clear to me exactly what's happening, but it looks like the modem just stops sending anything on its QMI endpoint for no reason.

How might I dive further into the issue? So far, my next step is to look
into CDC and QMI and try to decypher the USB traces. If anyone is interested, I can share a tcpdump or a USB trace.

I think a test of the backport on the same host hardware you run 3.6.2 on would be the best place to start.

Bjørn

#2 lore raw
Quoted 15 lines · Bjørn Mork
Quoted 10 lines · Shawn J. Goff

I've backported qmi-wwan to 2.6.39 (it's here:
https://bitbucket.org/accelecon/linux-at91/changesets), and it mostly works, but I've come across a problem. The modem will sometimes stop responding to any qmi data (but the AT commands on the TTY ports keep working). This only happens when there is significant traffic flowing through the device (downloading a large file) while at the same time, AT
commands are sent to one of the TTY ports (I first noticed with my own modem query program, but I can reproduce it using microcom to send "ATI\r" in a loop).

Using the tty ports should be completely independent of any qmi activity from the host perspective. I am tempted to claim this indicates a firmware bug.

Quoted 8 lines
Quoted 2 lines

I see this problem with different devices from
different manufacturers.

Which still may use pretty much the same firmware, although a little less likely.

Quoted 3 lines

I've only made it happen on my kernel - I
tried
on 3.6.2, but it seems to not happen there.

Good. But I have a feeling that you switched more than just the kernel. Do you see the issue if you run your backport on the same hardware you tested 3.6.2 on?

I just tested my 2.6.39 kernel on the same hardware that had 3.6.2; the problem is absent there.

Quoted 6 lines
Quoted 5 lines

I've also tried using a
similar modem that uses a different driver (sierra-net) and that doesn't
have the same problem.

Well, that is an entirely different firmware application and driver, even if the hardware is similar or even identical.

Yes - I wanted to eliminate anything lower (such as usb-net? not sure if qmi-wwan uses that) from being a suspect contributor to the problem.

Quoted 9 lines
Quoted 8 lines

When it is in failure, if I try to ping an address, the system sends out
several an ARP requests but gets no response. To get the device to respond again, I have to administratively set the wwan interface down, then up, use libqmi to get the connection going again, then dhcp to get

an address.

Which sounds like the connection died. Does QMI work at this point, or is that dead too?

Looks like qmi works. I can do --nas-get-signal-strength and it gives me good numbers. --wds-get-packet-service-status returns "Connection status: '2'"

Quoted 7 lines
Quoted 7 lines

I also have some USB traces of the failure and recovery process. I'm not
familiar with CDC or QMI, so it's not yet clear to me exactly what's happening, but it looks like the modem just stops sending anything on its QMI endpoint for no reason.

Quoted 7 lines
Quoted 4 lines

How might I dive further into the issue? So far, my next step is to look
into CDC and QMI and try to decypher the USB traces. If anyone is interested, I can share a tcpdump or a USB trace.

I think a test of the backport on the same host hardware you run 3.6.2 on would be the best place to start.

Bjørn

Thanks for your help.

#3 lore raw
Quoted 7 lines · Shawn J. Goff
Quoted 6 lines · Bjørn Mork
Quoted 3 lines · Shawn J. Goff

I've only made it happen on my kernel - I
tried
on 3.6.2, but it seems to not happen there.

Quoted 6 lines
Quoted 3 lines

Good. But I have a feeling that you switched more than just the kernel. Do you see the issue if you run your backport on the same hardware you tested 3.6.2 on?

I just tested my 2.6.39 kernel on the same hardware that had 3.6.2; the problem is absent there.

As I suspected. Then I believe this issue is more likely related to your hardware platform and/or its other lower layer drivers, and not to the backported qmi_wwan driver directly.

Maybe an obscure firmware issue related to timing or other differences? That is going to be difficult to track down, if it really is the cause.

Quoted 11 lines
Quoted 7 lines
Quoted 4 lines

I've also tried using a
similar modem that uses a different driver (sierra-net) and that doesn't
have the same problem.

Well, that is an entirely different firmware application and driver, even if the hardware is similar or even identical.

Yes - I wanted to eliminate anything lower (such as usb-net? not sure if qmi-wwan uses that) from being a suspect contributor to the problem.

Yes, that is useful. You are perfectly right that most of the host side drivers are common. Both sierra_net and qmi_wwan are usbnet minidrivers, and almost all network device functionality is served by the shared usbnet framework. So this test pretty much eliminates the host USB stack.

Which IMHO points to the firmware implementation as the major difference.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

When it is in failure, if I try to ping an address, the system sends out
several an ARP requests but gets no response. To get the device to respond again, I have to administratively set the wwan interface down, then up, use libqmi to get the connection going again, then dhcp to get

Quoted 6 lines
Quoted 2 lines
Quoted 1 line

an address.

Which sounds like the connection died. Does QMI work at this point, or is that dead too?

Looks like qmi works. I can do --nas-get-signal-strength and it gives me good numbers. --wds-get-packet-service-status returns "Connection
status: '2'"

Ah, interesting. Then we only need to find out where the bulk URBs end up.

I would have tried to enable what I could of USB and usbnet debugging to see if there are any hints there. A usbmon trace may also show the problem. I don't know. But in any case, I believe this is a USB problem and probably not too interesting for netdev...

Bjørn

#4 lore raw
Quoted 10 lines · Bjørn Mork
Quoted 9 lines · Shawn J. Goff
Quoted 8 lines · Bjørn Mork
Quoted 3 lines · Shawn J. Goff

I've only made it happen on my kernel - I
tried
on 3.6.2, but it seems to not happen there.

Good. But I have a feeling that you switched more than just the kernel. Do you see the issue if you run your backport on the same hardware you tested 3.6.2 on?

Quoted 8 lines
Quoted 2 lines

I just tested my 2.6.39 kernel on the same hardware that had 3.6.2; the problem is absent there.

As I suspected. Then I believe this issue is more likely related to your hardware platform and/or its other lower layer drivers, and not to the backported qmi_wwan driver directly.

Maybe an obscure firmware issue related to timing or other differences? That is going to be difficult to track down, if it really is the cause.

That's what I figured when I coudn't reproduce it on the other hardware. The first thing I'm going to do is check the power - I noticed the USB +5V supply can drop to around 4.8V when the modem is under load. I'll apply a bench power supply and see if that fixes things.

Quoted 29 lines
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

I've also tried using a
similar modem that uses a different driver (sierra-net) and that doesn't
have the same problem.

Well, that is an entirely different firmware application and driver, even if the hardware is similar or even identical.

Yes - I wanted to eliminate anything lower (such as usb-net? not sure if qmi-wwan uses that) from being a suspect contributor to the problem.

Yes, that is useful. You are perfectly right that most of the host side drivers are common. Both sierra_net and qmi_wwan are usbnet minidrivers, and almost all network device functionality is served by the shared usbnet framework. So this test pretty much eliminates the host USB stack.

Which IMHO points to the firmware implementation as the major difference.

Quoted 11 lines
Quoted 8 lines
Quoted 7 lines

When it is in failure, if I try to ping an address, the system sends out
several an ARP requests but gets no response. To get the device to respond again, I have to administratively set the wwan interface down, then up, use libqmi to get the connection going again, then dhcp to get

an address.

Which sounds like the connection died. Does QMI work at this point, or is that dead too?

Looks like qmi works. I can do --nas-get-signal-strength and it gives me good numbers. --wds-get-packet-service-status returns "Connection
status: '2'"

Quoted 7 lines

Ah, interesting. Then we only need to find out where the bulk URBs end up.

I would have tried to enable what I could of USB and usbnet debugging to see if there are any hints there. A usbmon trace may also show the problem. I don't know. But in any case, I believe this is a USB problem and probably not too interesting for netdev...

After power, I'll move on to enabling whatever USB debugging I can find. Thanks!

#5 lore raw
In reply to Shawn J. Goff
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

When it is in failure, if I try to ping an address, the system sends out
several an ARP requests but gets no response. To get the device to respond again, I have to administratively set the wwan interface down, then up, use libqmi to get the connection going again, then dhcp to get

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

an address.

Which sounds like the connection died. Does QMI work at this point, or is that dead too?

Looks like qmi works. I can do --nas-get-signal-strength and it gives me good numbers. --wds-get-packet-service-status returns "Connection
status: '2'"

Out of topic... I just fixed qmicli so that it prints the enum nickname string instead of the integer value.

Signature · 1 line
Aleksander